Hera Annual Review 2012 Final Report - Independent Monitoring, Review & Evaluation of WHO Prequalification Programme and Quality for Reproductive Health Medicines Programme

Publication date: 2012

Laarstraat 43, B-2840 Reet, Belgium tel +32-3-8445930 E-mail : hera@hera.eu fax +32-3-8448221 www.hera.eu Independent Monitoring, Review & Evaluation of WHO Prequalification Programme and Quality for Reproductive Health Medicines Programme Annual Review 2012 – Final Report 30 November 2012 Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 i Independent Monitoring, Review & Evaluation of WHO Prequalification Programme and Quality for Reproductive Health Medicines Programme Annual Review 2012 – Final Report 30 November 2012 Consultant Team: Wilbert Bannenberg (Team Leader) Marianne Schurmann Comments to: Wilbert@hera.eu and Marianne@hera.eu Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 ii TABLE OF CONTENTS List of Tables and Figures . iv Acknowledgements . iv List of abbreviations and acronyms . v Executive Summary . vii 1 Introduction . 1 2 Background and context . 2 2.1 Current need and funding for modern contraceptive methods . 2 2.2 Global reproductive health product markets . 3 2.3 The Reproductive Health Supplies Coalition . 4 3 Methodology . 4 4 WHO Prequalification Programme for Reproductive Health Medicines . 6 4.1 Summary of earlier reviews and evaluations . 7 4.2 Phase II Supplement review September 2011 to August 2012 . 8 4.2.1 Project documentation . 8 4.2.2 Progress towards achieving project objectives and goal . 9 4.2.3 Financial execution . 14 4.2.4 Business processes of the WHO Prequalification Programme for Medicines . 16 4.2.5 Project management . 18 4.2.6 Outcomes, challenges and lessons learnt . 19 4.2.7 Conclusions and specific recommendations . 20 5 Quality of Reproductive Health Medicines Programme (QuRHM) . 22 5.1 Introduction . 22 5.1.1 AQAS project 2009-2011 . 23 5.1.2 QuRHM project 2011-2014 . 23 5.2 Progress on outputs first 12 months . 25 5.3 Financial expenditure vs. budget . 27 5.3.1 Value for Money . 29 5.4 Efficiency of Concept Foundation processes per output area . 30 5.5 Concept Foundation internal business procedures . 32 5.6 Expert Review Panel (ERP) and other interim mechanisms . 32 5.7 Stakeholder management . 33 5.7.1 Transparency . 35 5.8 Technical Advisory Committee (TAC) . 36 5.9 Perceptions and comments of stakeholders . 37 5.10 Outcomes, challenges & lessons learned so far . 37 Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 iii 5.11 Conclusions and recommendations . 38 6 WHO PQP and Concept Foundation working together . 39 6.1 Structures for coordination. 40 6.2 Assessment of achievement of joint outcomes for 2012 . 41 6.3 Collaboration between the two programmes in practice . 42 6.4 Joint governance of WHO PQP/RH and QuRHM Programme. 43 6.5 Challenges and lessons learnt . 44 6.6 Conclusions and recommendations . 45 7 Risks, external factors and overall challenges . 45 7.1 PQ manufacturer not interested in specific markets, sectors or tenders . 49 7.2 Supply side focus . 49 7.3 Downstream risks . 50 8 Overall conclusions and recommendations . 51 8.1 Conclusions . 51 8.2 Overall recommendations . 54 ANNEXES . 55 Annex 1. Terms of Reference – Version 3 August 2012 . 56 Annex 2. List of Stakeholders interviewed . 63 Annex 3. Timetable . 67 Annex 4. Bibliography . 68 Annex 5. Pre-qualification Process Steps . 72 Annex 6. Revised Theory of Change Framework combining WHO PQP/RH and QuRHM Programmes 73 Annex 7. Annual Review 2013 and Impact Evaluation 2014/5 . 74 Annex 8. Questions for debate and next year’s Annual Review . 76 Why are so few generic hormonal contraceptives prequalified after 6 years?. 76 What do the manufacturers say? . 76 What is different in the RH medicines market compared to ARVs, TB, malaria, IUD and condom manufacturers? . 77 Are we too stringent? . 77 Can we get affordable, quality-assured APIs? . 78 Are the barriers technical or economical? . 78 Can we overcome the technical information gap? . 78 If there is no business case, then what? . 79 Can we get all donors and procurers to agree on one joint QA policy? . 80 Will prices of quality-assured generic products be lower? . 80 How long do we need the ERP process? . 81 Is WHO Prequalification there to stay?. 81 Are the non-PQ or non-SRA generic RH medicines bad quality? . 82 Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 iv If not generics, can we get lower prices from originator companies? . 82 Finally, are we pushing too much? . 82 Annex 9. Case Study Famy Care . 83 Annex 10. List of API and FPP manufacturers and their status. 84 Annex 11 - DFID Annual review template for QuRHM project . 87 LIST OF TABLES AND FIGURES Table 1: Comparison of expenditure (in US$) between 2009 and 2010 . 3 Table 2: Critical milestone status as per 27 August 2012 . 9 Table 3: List of RH products under assessment by WHO PQP (August 2012) . 10 Table 4: Overview of RH product related inspection activities and outcomes . 11 Table 5: Financial status BMGF funding of WHO PQP RH as per 30 June 2012 . 14 Table 6: Selected WHO PQP key performance indicators (actual values and targets) . 16 Table 7: Planned milestones & outputs vs. achievements per 31 July 2012 (12m QuRHM) . 25 Table 8: DFID output ratings . 26 Table 9: QuRHM Programme budget . 27 Table 10: Status of joint outcomes for WHO/PQP RH and QuRHM programmes . 42 Table 11: Assessment of QuRHM programmes assumptions . 46 Table 12: Recommendations on QuRHM risk rating modification . 47 Table 13: Assessment of WHO PQP/RH assumptions . 48 Figure 1: Time line of RHSC quality activities, 2005-2011 . 22 Figure 2: Concept Foundation expenditures vs. budget FY1 (GBP) . 27 Figure 3: Concept Foundation expenditures vs. budget first half FY2 (GBP) . 28 Figure 4: Concept Foundation expenditures vs. budget from project start up to 30 Sept 2012 (GBP) 28 Figure 5: WHO/PQP and QuRHM Programmes working together . 40 Figure 6: Examples for supply and demand side criteria providing incentives for prequalification . 50 ACKNOWLEDGEMENTS This report is developed under contract from the Bill and Melinda Gates Foundation (BMGF), in close collaboration with the Department for International Development (DFID) of the United Kingdom. The Review Team would like to express its gratitude to all officials and individuals of BMGF, DFID, RHSC, WHO, Concept Foundation and other partners who provided information and who graciously gave their time and support to the review process. Special thanks go to Leo Devillé for internal, and Cheri Grace for external quality assurance under time limited constraints. The Review Team, Cruïlles/Ulaanbaatar/Bergeijk, 30 November 2012 Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 v LIST OF ABBREVIATIONS AND ACRONYMS API Active Pharmaceutical Ingredient AQAS Accessing Quality Assured RH Supplies AR Annual Review BMGF Bill and Melinda Gates Foundation (www.gatesfoundation.org) BMZ Federal Ministry for Economic Cooperation and Development (www.bmz.de/en/) CAPA Corrective and Preventative Action(s) CEO Chief Executive Officer CF Concept Foundation (www.conceptfoundation.org/) cGMP Current Good Manufacturing Practices CHAI Clinton Health Access Initiative (http://www.clintonfoundation.org/main/our- work/by-initiative/clinton-health-access-initiative/about.html) COC Combined Oral Contraceptives CPhI Conference of the Pharmaceutical Industry (www.cphi.com) CPR Contraceptive Prevalence Rate CRO Contract Research Organisation CTD Common Technical Document DAC Development Assistance Committee DFID Department for International Development (www.dfid.gov.uk) DKT DKT International (www.dktinternational.org) DMPA Depot Medroxyprogesterone Acetate EC Emergency Contraceptive EMA European Medicines Agency (www.ema.europa.eu) EOI Expression of Interest ERP External Review Panel FDA Food & Drug Authority FHI 360 Family Health International (www.fhi360.org) FPP Finished pharmaceutical product GBP Great Britain Pound (£) GFATM Global Fund for AIDS TB and Malaria (www.theglobalfund.org) GMP Good Manufacturing Practices HC Hormonal Contraceptives ICDRA International Conference of Drug Regulatory Authorities (www.icdra.ee) ICH International Conference on Harmonisation (www.ich.org) IPPF International Planned Parenthood Federation (www.ippf.org) IUD Intra-uterine device JSI John Snow International (www.jsi.com) KfW Kreditanstalt für Wiederaufbau (www.kfw.de/kfw_/kfw/en/) KPI Key Performance Indicator LPV/r Lopinavir + ritonavir MDA (WG) Market Development Approaches (Working Group) MOH Ministry of Health MMR Maternal Mortality Ratio MoU Memorandum of Understanding MSI Marie Stopes International (www.mariestopes.org) M&E Monitoring & Evaluation NMRA National Medicines Regulatory Authority OC Oral contraceptive OHC Oral hormonal contraceptives Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 vi PIC/S Pharmaceutical Inspection Cooperation Scheme (www.picscheme.org) POC Programme Oversight Committee PQ Prequalified PQP/RH Prequalification Programme for Reproductive Health Medicines PQP Prequalification Programme (www.who.int/prequal) QA Quality Assured QCL Quality control laboratory QMRH Quality Medicines for Reproductive Health (BMGF project with WHO PQP) QSM Quality & Safety of Medicines (WHO unit) QuRHM Quality of Reproductive Health Medicines (DFID project with RHSC/CF) RH Reproductive Health RHSC Reproductive Health Supplies Coalition (www.rhsupplies.org) RT Review Team SRA Stringent Regulatory Authority SS (WG) Systems Strengthening (Working Group) TAC Technical Advisory Committee TOR Terms of Reference UNFPA United Nations Population Fund (www.unfpa.org) UNFPA-PSB United Nations Population Fund – Procurement Services Branch (www.unfpa.org/public/procurement) UNICEF United Nations Children’s Fund (www.unicef.org) UNITAID (www.unitaid.eu) USAID United States Aid agency (www.usaid.gov) USD United States Dollar ($) USP United States Pharmacopoeial Convention (www.usp.org) VfM Value for Money (www.dfid.gov.uk/Documents/publications1/DFID-approach-value- money.pdf) WG Working group (of RHSC) WHO World Health Organization (www.who.int) Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 vii EXECUTIVE SUMMARY The international community has embraced the issue of poor access to good quality hormonal contraceptives and other RH medicines. The current institutional public sector market is dominated by originator products that are relatively costly, especially implants and injectables. More affordable generic products are available but they have not always undergone a rigorous assessment of cGMP and registration dossiers, preventing them from being procured by international donors. BMGF has been supporting the WHO PQP since 2006 to establish an independent and rigorous assessment process of RH medicines. The number of prequalified generic RH medicines has however remained disappointingly low. The prevailing view has been that technical barriers, rather than economic challenges, were the primary basis for the low number of pre-qualified products. This motivated DFID to start a parallel programme of technical support of generic manufacturers through RHSC and implemented by the Concept Foundation. BMGF and DFID joined hands, and decided to request 2 joint annual reviews of both programmes in 2012 and 2013, and an impact assessment in 2014/5. This report is the first in the series, and apart from a standard review of progress, focuses on the manufacturers’ and supply side. It should be noted that the BMGF and DFID projects were developed independently; the subsequent alignment of targets adds complexity. The hypothesis behind both projects is that an increased number of quality assured generic products can create more competition between quality-assured product manufacturers, lower prices, offer more choice to procurers and women, and satisfy unmet need. WHO PQP has worked hard to make its PQP for RH a well-run operation. It is doing a good technical job and adheres to SOPs and KPIs, while being open for criticism and trying to address the issues raised where possible. WHO PQP has regular meetings with manufacturers, but could improve its communication with other stakeholders, and proactively promote its programme. “PQ needs to be demystified.” WHO also tries to build capacity in national Medicines Regulatory Authorities in low- income countries, and invites their staff to participate in GMP inspections and dossier assessments. Despite the hard work, the number of PQ generic RH medicines has remained low. The disappointing PQ results were reason for DFID to embark on QuRHM, a project run through RHSC and implemented by Concept Foundation to strengthen the technical capacity of generic RH manufacturers, to promote a harmonised Quality Assurance policy in procurement among donors and procurers, to advocate the procurement and use of quality assured RH medicines at country level, and to increase general awareness and advocacy of the programme inside RHSC. The programme design was focused primarily on the supply (push) rather than the demand (pull) side, probably due to the core business aspects of the requesting organisations WHO/PQP and Concept Foundation. Concept Foundation appears to be making steady progress in its TA work with generic companies, although insight on details on these engagements was not offered to the RT. Concept Foundation is optimistic about the speed by which it can support generic companies and deliver them to the WHO PQP for assessment. The four generic companies, who volunteered information to the RT on Concept Foundation’s work, are generally satisfied about the support they get. Concept Foundation has been giving less attention to the downstream activities, which it has contracted out to UNFPA and consultants. RHSC should probably take more responsibility here, especially in the work in/with countries. Assessing value for money (VfM) has been a challenge as Concept Foundation was not asked by DFID in its contract to assess/monitor VfM. Over the review Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 viii period, Concept Foundation’s capacity to deliver or supervise all outputs was stretched. Internal management is now being strengthened, so that Concept Foundation can properly supervise its outsourced activities. The subcontract with UNFPA could be clearer in describing outputs; efficiency and VfM were difficult to assess at this point as the 1st financial report by UNFPA was not yet available, and RT requests for specific financial details were not answered by UNFPA with reference to the UN rules of the “single audit principle”1. The ERP process was helpful in generating 12 more quality assured products for procurers. Concept Foundation’s stakeholder management can be strengthened, and overall responsiveness improved. It could also make better use of the Technical Advisory Committee, which was supposed to ensure links with the wider RHSC community. Donors DFID and BMGF made a good decision to have the two programmes work together. WHO PQP and Concept Foundation’s work is complementary, and both are needed to get more quality assured RH medicines to the market. The coordination structures could work more effectively, and most of the good collaboration was achieved by ad-hoc visits and personal relations. A joint monitoring plan has just been developed and it was too early to see results. A joint Programme Oversight Committee (POC) meets regularly and enthusiastically, but makes largely ad-hoc decisions and does not follow-up previous minutes or provides structural supervision of the project. Reviewing both programmes, it is still too early to tell whether the hypothesis (that the stimulation of more RH medicines being prequalified or ERP recommended will stimulate competition, reduce prices, increase choice of quality assured products for procurers and women, and reduce unmet need) will work. However, the pipeline contains some much needed products, and the RT suggests that both organisations should be given more time to deliver their outputs. They have good technical skills for the regulatory and TA work, but less so for some of the downstream issues. Their output/milestone rating for year 1 is lower than expected due to delays in contract signing, and an overly ambitious timetable in the project design. Results of the forthcoming business case work should provide an improved evidence base upon which to better focus and tailor the grantees’ efforts towards those manufacturers that have the most potential for market shaping. The report provides some suggestions of areas in which both WHO PQP and Concept Foundation can strengthen their systems and communications, but the RT is confident that both can deliver the outputs when given more time and with an enhanced evidence base to help tailor efforts. The RHSC should take more responsibility for the work in/with countries, and donors should contemplate how they can involve RHSC to also cover the downstream demand side in the coming years. 1 UNFPA (and all other UN organizations) are operating under the single Audit principle: http://www.un.org/Depts/oios/pages/audit%20manual%20-%20march%202009%20edition.pdf Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 1 1 INTRODUCTION Unmet demand for family planning and other reproductive health commodities remains unacceptably high, especially in low- and middle-income countries. Complications related to unintended pregnancies greatly contribute to overall newborn, infant and maternal mortality rates. Improving access to affordable quality assured essential reproductive health commodities is thus a key intervention for achieving Millennium Development Goals 4 and 5. Low- and middle-income countries’ public sector markets for hormonal contraceptives with assured quality in terms of international standards (i.e. either WHO prequalified or Stringent Regulatory Authority (SRA) registered) have been dominated by originator companies selling their products at preferential prices for use in low- and middle-income countries’ public sector programmes. Increasing the number of manufacturers that offer quality assured products is expected to (i) lead to price reductions due to competition, and (ii) address the risk of overall product shortages existing in oligopolistic markets (unwillingness to increase production capacity; market exit). However, a report of two studies conducted in 2006 concluded that less than 30% of 44 pharmaceutical companies in 15 countries producing hormonal contraceptives met stringent Good Manufacturing Practices (GMP) requirements, and few had developed adequate registration dossiers.2 In this context the Bill & Melinda Gates Foundation (BMGF) and the Department for International Development (DFID) are funding two complementary programmes aiming at improving availability and choice of quality assured low-cost reproductive health commodities (focus on hormonal contraceptives) in low- and middle-income countries:  BMGF supports the WHO Prequalification Programme for Reproductive Health (PQP RH). The programme goal (as per framework table) is to have an expanded selection of quality assured and approved RH medicines resulting in increased contraceptive availability and choice in developing countries.  DFID supports the Quality for Reproductive Health Medicines (QuRHM) programme implemented on behalf of the Reproductive Health Supplies Coalition (RHSC) by Concept Foundation. QuRHM has been designed to address contextual issues that currently prevent manufacturers from either applying to or successfully completing the WHO prequalification process. The overall project hypothesis is that “increased availability of quality RH products will lead to price reductions and greater volumes procured. The cost savings as a result are applied to further procurement, hence increasing overall supply and reducing unmet need, contributing to reducing the number of unintended pregnancies in low- and middle-income countries. In addition, major procurers and countries will have a wider choice of quality assured and approved suppliers, who can manufacture quality products at lower prices. This should also lead to improved method mix available in country.” (See Annex 1: Terms of Reference). The BMGF and DFID have contracted HERA to do two independent external annual organisational and programme reviews (in 2012 and 2013), and an impact evaluation (in 2014/15) of the two programmes. The subject of this document is the first annual review that was conducted during the period July to October 2012. 2 Hall P, Oehler J et al. A study of the capability of manufacturers of generic hormonal contraceptives in lower- and middle-income countries. Contraception 75 (2007) 311 – 317. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 2 During the inception phase discussions were held with the two funders and the two implementing partners. These resulted in a revised version of the original Terms of Reference where specific questions for the Annual Review 2012 were added (see Annex1: Terms of Reference, 2.e). The report begins with an overview of the background and context within which the two programmes operate followed by a description of the methodology used for the annual review. Progress is then assessed separately for each programme followed by the review of how the two programmes worked together to achieve their common goals. External factors impacting progress have been assessed and a set of emerging critical questions addressed. The report ends with a set of overall conclusions and recommendations. 2 BACKGROUND AND CONTEXT Both programmes address the need for quality assured affordable reproductive health medicines which in this context are those included in the respective WHO PQP Expressions of Interest. Currently these are 7 oral hormonal contraceptives (including 2 emergency contraceptives), 4 injectable hormonal contraceptives, 2 implantable contraceptives, 3 oxytocics, and one medicine for prevention and treatment of eclampsia (magnesium sulphate). Relatively recently the scope of reproductive health medicines was expanded beyond contraceptives, but much of the current discussion around reproductive health medicines still focuses on hormonal contraceptives. 2.1 Current need and funding for modern contraceptive methods It is estimated that in 2012 222 million women in low- and middle-income countries are having an unmet need for modern contraceptive methods, 162 million of them living in the world’s poorest countries.3 Providing these 222 million women with modern contraceptive methods would prevent 54 million unintended pregnancies, and avert 79,000 maternal and 1.1 million infant deaths at an estimated total cost of USD 2.08 billion. Out of this amount USD 573 million would be spent on contraceptive commodities alone, representing a more than 50% increase of the amount estimated to be currently spent to provide contraceptive commodities to existing users (USD 947 million)4. Funding sources for family planning commodities vary widely per geographical region. A study conducted on behalf of the RHSC in 2009 found that in the African region more than 80% of supplies are donor funded, while the donor funded share in Latin America and the Caribbean is 33%, and in the Middle East/North Africa and Asia/Pacific regions and the Central Asian Republics 20-25%. Additional funding sources are government budgets and private out-of-pocket expenditure.5 Donor contributions to modern contraceptives purchases are documented annually by UNFPA. The most recent figures are reproduced in Table 1 below6. UNFPA and USAID are major funders, having a 35 and 32 percentage share of the total donor funding in 2010. KfW and PSI follow with 12 and 11 percentage share, while DFID had a 7 percentage share. 3 Modern methods include male and female sterilization, IUDs, implants, injectables, oral hormonal contraceptives, male and female condoms and other supply methods, such as spermicides; Poorest countries: GNP/capita of ≤ USD 2,500 in 2010 (n=69). 4 Singh S and Darroch JE. Adding it up: Costs and benefits of Contraceptive Services – Estimates for 2012. Guttmacher Institute and UNFPA, New York 2012; http://www.guttmacher.org/pubs/AIU-2012-estimates.pdf 5 Ross J et.al. Contraceptive Projections and the Donor Gap – Meeting the Challenge; Reproductive Health Supplies Coalition; Brussels: 2009 6 UNFPA: Donor Support for Contraceptives and Condoms for Family Planning and STI/HIV Prevention 2010; New York: 2011; www.unfpa.org/public/home/publications/pid/6479 Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 3 Table 1: Comparison of expenditure (in US$) between 2009 and 2010 2009 2010 Comparison 2009-2010 Change Percentage DFID 13,005,195 16,590,831 3,585,636 27.57 BMZ/KfW 16,189,032 29,180,788 12,991,756 80.25 PSI 17,942,658 26,909,321 8,966,663 49.97 UNFPA 81,136,535 82,391,543 1,255,008 1.55 USAID 87,549,507 76,014,739 (11,534,768) -13.18 IPPF & MSI 22,977,954 4,075,792 (18,902,162) -82.26 Total 238,800,882 235,163,014 (3,637,868) -1.52 The publication does not comment on the impact of the decrease in spending from 2009 to 2010, e.g. whether this signifies that less volumes were procured or that the same (or more) volume was procured at lower cost. The same publication provides information on the share of expenditure (from these funders only) per commodity and region: In 2010  32% was spent on male condoms, 24% on hormonal injectables, 22% on hormonal oral contraceptives, 14% on implants, and 8% on other methods;  In terms of commodity value: 63% was allocated to Africa, 25% to Asia and Pacific Islands, 7% to Latin America and the Caribbean, and 4% to the Arab States. At the Family Planning Summit held in London in July 2012 the global community committed to sustaining existing levels of modern contraceptives use and provide additional resources to address unmet need by providing family planning services and commodities to an additional 120 million women in the 69 poorest countries by the year 2020. The cumulative costs for achieving this goal were estimated at approximately USD 4 billion out of which USD 2.5 billion would be needed for the provision of contraceptive commodities7. It was estimated that USD 2.3 of the total 4 billion would need to be sourced from donor funds, and by September 2012 donors’ commitments amounted to USD 2.6 billion (including new commitments made in 2010 for the period 2012-2015)8. 2.2 Global reproductive health product markets It is important to note that the table above provides only a snapshot of a segment of the market – the donor financed portion. We do not have total market data with which to determine if the above 235 million represents 25%, 50% or 75% of the total market, therefore it is difficult to gauge the relative importance of activities, such as this programme, which seek to shape the supply and demand of this segment. Comprehensive information on the global reproductive health product market, including both public and private sectors, could not be sourced. Related research results are available in the private sector but at a high cost.9 7 Anonymous. London Summit on Family Planning, July 2012: Technical Note: Data sources and methodology for developing the 2012 baseline, 2020 objective, impacts and costings – Working Draft – 15 June 2012 8 London Summit on Family Planning: Summary of Commitments; www.londonfamilyplanningsummit.co.uk/COMMITMENTS_090712.pdf 9 E.g. at: http://www.strategyr.com/Contraceptives_Market_Report.asp Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 4 2.3 The Reproductive Health Supplies Coalition RHSC was created in 2004, and describes itself as “a global partnership of public, private and non- governmental organizations working together to ensure that all people can choose, access and use affordable, high-quality reproductive health products”. By October 2012 RHSC consisted of 203 members including multi- and bi-lateral organisations, private foundations, low- and middle-income country governments, non-governmental organisations, and the private sector10. Core support is currently being provided by BMGF, DFID, UNFPA and USAID. An evaluation of the RHSC conducted in 201211 concludes that achievements against the RHSC strategic plan objectives at global level have been overall positive, while less progress has been made at regional and particularly country level where activities started more recently. Amongst global level strategies to increase reproductive health commodity security the RHSC’s efforts to promote the quality agenda for RH medicines (including prequalification) are mentioned. The comparative advantage of the RHSC as providing a neutral space where different stakeholders with diverse interests convene was confirmed. 3 METHODOLOGY The methodology described in the Inception Report had to be slightly adapted, mainly related to consultation processes with manufacturers. Evaluation framework. HERA used the OECD Development Assistance Committee (DAC) evaluation criteria (relevance, efficiency, effectiveness, impact and sustainability) with related indicators, definitions and evaluation tools.12 Project document review. We first reviewed all available technical, project and policy documents from BMGF, DFID, WHO, Concept Foundation and RHSC websites related to the subject before requesting detailed project documents from BMGF and DFID. Additional documentation was then requested and received from WHO PQP and Concept Foundation: grant applications, work plans, budgets, financial and progress reports. Internal reports and assessments of work with generic RH manufacturers were not accessible with a reference to the confidentiality contracts signed. Value for Money reports did not exist as this requirement was not included in the original contracts with WHO/PQP and Concept Foundation, so the reviewers had to rely on available budgets, unit costs and financial reports to judge Value for Money in the projects. On specific request by the Review Team (RT), WHO PQP kindly developed financial and progress interim reports as per 30 June and 17 August 2012 respectively, as their last routine report to BMGF covered the period until 31 December 2011 only. Concept Foundation provided the QuRHM progress report developed for the October 2012 RHSC Conference in Paris covering the first 12 programme months from 1 August 2011 to 31 July 2012. PATH and Concept Foundation also provided updated spending information. External information review. HERA searched websites and approached contacts in key stakeholder organizations essential for meeting program objectives, e.g. RHSC members, Secretariat and Market 10 www.rhsupplies.org 11 Lauro D, Chattoe-Brown A. Evaluation of the Reproductive Health Supplies Coalition. Final Report. 30th June 2012; www.rhsupplies.org/fileadmin/user_upload/Evaluation_Report/Evaluation_2012.pdf 12 Glossary of Key Terms in Evaluation and Results Based Management OECD, 2010. http://www.oecd.org/development/peerreviewsofdacmembers/2754804.pdf Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 5 Development Approaches and Systems Strengthening working groups, donors (UNFPA, USAID, World Bank), social marketing organizations (FHI 360+, DKT, PSI, MSI) and international procurement agencies (UNFPA PSB, JSI). We also searched for RH medicines quality information on the websites of National Medicines Regulatory Authorities (NMRAs) in the 4 pilot countries. Client and partner’s visits. HERA visited the client offices (BMGF Seattle, DFID London) and the contracted partners (PATH, WHO PQP, Concept Foundation, UNFPA PSB), and interviewed project staff and managers. Follow-up discussions were held by email and telephone, and during subsequent conferences (Paris, Copenhagen and Tallinn). Interviews with key stakeholders were based on semi- structured questionnaires. Manufacturer’s interviews. The RT did not have access to a list of generic manufacturers supported by CF and/or in the WHO PQP pipeline. Concept Foundation provided a list of all RH medicines manufacturers they had ever identified, without indicating whether the listed companies were being (or had been) supported by them. This list was supplemented with RH manufacturers identified through own research, and those participating in the UNICEF/ WHO PQP manufacturers meeting in Copenhagen, the RHSC conference in Paris, and the Global Pharmaceutical Industry conference (CPhI) in Madrid. Interviews were held face to face at these three meetings. Additional interviews were done by phone. All manufacturers’ representatives were assured that their contributions would be kept confidential and be reported without mentioning the respondent’s individual or company name. As the RT did not have access to the denominator, no rates can be presented. Regulatory Authorities interviews. HERA interviewed medicines’ regulators of lower income countries during the 21-26 October 2012 global ICDRA conference in Tallinn, Estonia. These included the 3 NMRAs of the 4 pilot countries that attended the conference. As requested, the QuRHM programme baseline data collection and pilot programme was not mentioned to these countries. A list of all stakeholders interviewed organised by stakeholder group is attached in Annex 2. Analysis. We consolidated information received through document review and interviews with a focus on triangulation of the data and evidence generated through programme monitoring. We considered the progress of the programmes with particular regard to the programme outputs; the quality and value of those outputs (also considering the technical assistance recipients’ views); and the ways in which the outputs were shaping the programme outcome. We identified how new risks had been managed or mitigated; analysed the roles of the main stakeholders and how these contributed to the programme. We assessed the complementarities of the WHO PQP and Concept Foundation activities, in the context of other global efforts to promote quality production, registration and regulation of RH products, and whether complementarity was ensured. At the clients’ request, specific attention was given to:  Whether the governance and management arrangements are appropriate and effective.  Effectiveness of the communication strategy between the two programmes and with all other stakeholders.  Effectiveness of current reporting material.  Lessons learned – how lessons and best practices are being disseminated.  Coherence and coordination – to what extent the programmes are consistent with and inform other activities/agencies to promote access to good quality RH products.  The extent to which the desired outputs have been achieved with the lowest possible use of resources.  Sustainability - the extent to which the benefits of the project will be continued after the completion of the project or the end of the funding Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 6 Reporting. The draft narrative report was shared with the clients and their contracted partners Concept Foundation and WHO PQP. Extensive written feedback was provided by the stakeholders individually, and the RT had access to the POC meeting minutes were the draft report had been discussed jointly. Based on this feedback the RT contacted the relevant stakeholders for issues for clarification, and corrected factual errors. Other comments were taken on board where the RT felt these were justified and within the scope of this review. As required in the Terms of Reference the RT was also guided by maintaining the independent and external character of the review Value for Money. VfM reports could not be analysed by us as these were not routinely generated by Concept Foundation or PATH (no contractual requirement). We therefore could only assess the budgeted amounts utilised against the project outputs, and unit costs and financial reports (if available). Case study. Two Indian generic manufacturers of RH products, CIPLA and Famy Care, were approached to see whether they were willing to be interviewed for an additional case study. Only Famy Care agreed to be interviewed for the case study. The draft case study report will be presented in December 2012 after on-site interviews of Famy Care staff in India. Impact Assessment preparation. HERA has not been able to make much progress on the 2014/15 impact assessment preparations, as the necessary baseline data collection by Concept Foundation had not yet started. HERA commented on the draft RFP, and will review the outputs of the country baseline studies in March 2013. Other Challenges. Concept Foundation did not provide the RT access to the list of manufacturers supported by them. The RT could therefore not systematically contact manufacturers for interviews, or assess the efficiency of Concept Foundation’s work with manufacturers. Amongst the manufacturers directly approached by the RT some were receiving support from Concept Foundation. They had no hesitation discussing this or to provide feedback. For the annual review in 2013 the RT proposes to sign a confidentiality agreement between with Concept Foundation that will facilitate identifying and contacting the relevant manufacturers, and assessing Concept Foundation’s efficiency. 4 WHO PREQUALIFICATION PROGRAMME FOR REPRODUCTIVE HEALTH MEDICINES BMGF has funded the “Quality Medicines for Reproductive Health (QMRH)”13 programme implemented by WHO and partners since 2004:  Phase I was completed in 2006 and focussed on harmonised and evidence based selection of essential medicines for reproductive health.  Phase II (total grant amount USD 3.49 million) was implemented from August 2006 to September 2010 (3 years plus a 12 months no-cost extension). The overall programme objective was to improve access to quality essential reproductive health medicines and commodities, and the 4 focus areas were (i) prequalification of reproductive health medicines, (ii) building of procurement capacity at country level, (iii) harmonisation of quality standards for reproductive health products of UN agencies and NGOs, and (iv) establishment of a reproductive health quality resource centre. USD 1.23 million of the total grant amount 13 Not to be confused with the “Quality for Reproductive Health Medicines” (QuRHM) programme started in 2011 Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 7 was dedicated to work related to prequalification. The grant was managed by WHO Department of Essential Medicines and Pharmaceutical Policy and implemented with two main partners, WHO Department of Reproductive Health and Research and PATH (as sub- grantee).  The Phase II Supplement grant (USD 1.49 million) was awarded in September 2011 and runs over a period of 15 months up to end December 2012. During this period the programme is focussing on prequalification of generic reproductive health medicines and Active Pharmaceutical Ingredients (APIs), and contextual issues to improve the rate of submissions to prequalification and time to approval. The grant is managed by WHO Department of Essential Medicines and Pharmaceutical Policy/Quality and Safety: Medicines. This grant is the subject of the current annual review. BMGF is currently considering a new grant to WHO to support WHO PQP overall. 4.1 Summary of earlier reviews and evaluations The progress and milestone reports provided by WHO to BMGF under Phase II of the Quality Medicines for Reproductive Health (QMRH) programme mention a midterm and an end of programme review which however, had not been seen by the funder. In general the funder was disappointed about the slow progress with regard to prequalification: by the end of the original Phase II only 8 products had achieved WHO prequalification status, all of them with previous SRA approval and from Western manufacturers. BMGF would have appreciated a clear analysis by WHO as to why progress was slow. The WHO PQP Business Plan 2009-201314 provides some recommendations to increase efficiency and effectiveness of the Programme such as establishment of a steering committee for providing strategic direction and improving transparency; enhanced communication with donors and procurement agencies; institution of an anonymous feedback mechanism for manufacturers; improvement of internal monitoring systems including tracking of process indicators; and improvement of financial management systems. WHO PQP has started addressing these recommendations focussing on financial accountability and reporting. In 2010 WHO PQP in the context of a process optimisation review and in collaboration with a market research firm conducted a survey of manufacturers that successfully had completed the prequalification process between 2006 and 201015. The survey was designed to assess service processes (how the prequalification system is designed) and service delivery, and to provide information on the service level desired by clients and the service level provided by WHO PQP in practice. The main findings were that in general WHO assessors and inspectors met or exceeded respondents’ expectations in terms of service level. The same applied to the service process overall. Areas for process improvement included: in-person communication and problem resolution during assessment; consistency of team members during assessment process; stringency of WHO PQP GMP requirements; and the time required to prequalify a product. On behalf of UNITAID an external mid-term review of the UNITAID funded work of WHO PQP was conducted in early 201116. The overall finding was that WHO PQP was most likely to meet the 14 Price Waterhouse Coopers: Business Plan for WHO Prequalification of Medicines Programme 2009-2013. 7 August 2009 15 WHO Prequalification of Medicines Programme: Survey of service quality provided to manufacturers. WHO Drug Information Vol. 24, No. 4, 2010. www.who.int/medicines/publications/druginformation/issues/DrugInformation2010_Vol24-4/en/index.html See also http://apps.who.int/prequal/info_general/documents/2010_Manufacturers-Survey.pdf 16 Pouget C. Mid-term review of the UNITAID funded WHO Prequalification Programme; UNITAID / AEDES 2011; www.unitaid.eu/images/projects/Prequalification/20110617_MTR-WHO_PQP-FinalReport.pdf Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 8 defined objectives of the UNITAID programme (in terms of number of newly prequalified priority products and in terms of capacity building). A SWOT analysis was performed. Weaknesses identified included long time to reach prequalification (with the comment that much of this is beyond WHO’s area of influence); lack of appropriate data management systems; insufficient mechanisms for communication with stakeholders other than manufacturers. Opportunities included: easier monitoring of KPIs facilitated by a data management system that was under development; planned extension of capacity building activities; and the planned development of a business case for prequalification. Threats were identified as dependency on donor financing and difficulty in maintaining adequate human resources capacity due to WHO recruitment rules. A regulatory scan17 developed by McKinsey on behalf of BMGF presented the following challenges from the viewpoint of Indian and Chinese manufacturers of hormonal contraceptives (information on sample size or representativeness not available): not being clear about requirements for participation in the prequalification process and about WHO GMP guidelines; perception of WHO PQP inspectors being inconsistent and GMP guidelines being more stringent than US FDA; different requirements of regulatory authorities (and WHO PQP) in terms of documentation of quality, safety, efficacy; insufficient engagement with manufacturers during the assessment process. Reviews of WHO PQP were also conducted on behalf of the WHO Director General and in the context of the WHO Department of Reproductive Health and Research during 2011 and 2012 but these are not (yet?) public and authorisation to receive copies was not provided. 4.2 Phase II Supplement review September 2011 to August 2012 4.2.1 Project documentation None of the WHO PQP Quality Medicines for Reproductive Health (QMRH) programme documents (i.e. Phase II & Supplement) included a logical framework with clearly stated impact, outcome, outputs, inputs and related indicators, milestones and targets. While the Supplement Proposal (Narrative Support to Supplement Request dated 19 August 2011) contains a ‘Revised Project Framework Table’, listing for each of the two objectives ‘Indicators of Success’ and how they will be monitored, the Milestone Reporting Chart used for reporting to BMGF has maintained the original Phase II structure reporting on ‘critical milestones’ at objective level, and ‘activity milestones’ at activity level (44 milestones for 17 activity areas). There are no indicators for goal level in either case. The planned activities in both documents are identical. We understand that the Revised Project Framework Table is applicable to the supplement grant and making the appropriate changes to the original project templates was omitted unintentionally. We will therefore assess progress against objectives and goal in the supplement proposal:  Project goal: Expanded selection of quality assured and approved RH medicines resulting in increased contraceptive availability and choice in developing countries  Objective 1: Increase the availability of affordable, quality-assured RH medicines for developing countries  Objective 2: Coordinate activities with Concept Foundation on prequalification of RH medicines There are 7 indicators of success. 17 Bill & Melinda Gates Foundation: Regulatory Scan – Highlights. January 2012 Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 9 The Narrative Support for Supplement request is complemented by a ‘Budget narrative: supplement request Quality medicines for reproductive health (QMRH) project – WHO Prequalification of Medicines Programme (PQP)’, which provides the assumptions and justification for the supplement grant budget. Assumptions, such as daily rates for contracted experts are reasonable. Per diems and travel policy are as per WHO guidelines. WHO PQP also makes efforts to organise inspections more efficiently, e.g. by having inspections back to back or linking them with country workshops. 4.2.2 Progress towards achieving project objectives and goal Progress towards meeting the project’s critical milestones defined in the milestone reporting chart is documented in Table 2 below. Table 2: Critical milestone status as per 27 August 2012 Milestone Status Objective 1: Increase the availability of affordable, quality-assured RH medicines for developing countries 1. First WHO prequalification of a generic contraceptive by 12/2011 Achieved: 1 COC each prequalified in September and December 2011 2. First accelerated approval of WHO prequalified generic medicine by 03/2012 Delayed: revised due date 12/2012 Objective 2: Coordinate activities with Concept Foundation on prequalification of RH medicines 1. Planning and monitoring meeting between DFID, Gates and Concept Foundation by 11/2011 Achieved: Initial meeting held Nov/2011 (POC); follow-up meetings in February and July 2012 2. Joint monitoring and evaluation plan developed by PQP and Concept Foundation by 11/2011 Not achieved: a draft M&E plan was under development by Concept Foundation but not yet being implemented As per TOR and discussion with the two funders the annual reviews should report on progress on results rather than output level. Progress with regard to the 14 activity areas under Objective 1 are therefore summarised below using the 7 main activity areas from the supplement proposal. Specific indicators of success from the Project Framework Table are allocated to the main activity areas as appropriate. Progress on Objective 2 will be addressed in more detail in chapter 6 (Joint activities of WHO/PQP and Concept Foundation). 1) Norms and Standards that underpin WHO prequalification Indicator Target 12/2012: 2 specifications developed and 5 under development The specifications for Medroxyprogesterone Injection and Levonorgestrel + Ethinylestradiol Tablets18 were discussed at the Expert Committee meeting in October 2012. The specification for medroxyprogesterone injection was adopted, whereas the specification for levonorgestrel + ethinylestradiol tablets requires further development and will be reconsidered at the October 2013 meeting of the Expert Committee. The remaining APIs (3) and dosage forms (5) are included in the current work plan of the International Pharmacopoeia but drafts are not yet publicly available. 18 See: http://www.who.int/medicines/areas/quality_safety/quality_assurance/projects/en/index.html (accessed 1 Nov 2011) Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 10 2) Prequalification activities Indicator Targets 12/2012:  5-8 generic (or low-cost) contraceptives prequalified o 5 oral hormonal contraceptives (OHC) manufacturers o 1 implant manufacturer o 1 oxytocic manufacturer  10 dossiers submitted and 50% of these passed initial screening  At least one supplier for each of the product categories 1-419 of the current EOI for prequalification of RH medicines in the WHO prequalification pipeline (i.e. having a product being assessed) Two generic OHCs from two Indian manufacturers are prequalified, while 3 were under assessment (dossier submission in 2007 [1] and 2009 [2] respectively). In addition, one generic implant and 3 oxytocics were under assessment plus one innovator injectable. This represents at least one product per category. However, applications for 6 of the 10 pipeline products were received before 2011, and a generic Medroxyprogesterone injectable – one of the most needed products – is not included. The status of all RH products under assessment is documented in Table 3. Table 3: List of RH products under assessment by WHO PQP (August 2012) Product name Category Type Date received Ethinylestradiol + levonorgestrel COC Generic Sep 2009 Ethinylestradiol + levonorgestrel COC Generic Nov 2007 Ethinylestradiol + levonorgestrel (+ placebo) COC Generic Sep 2009 Levonorgestrel 1.5 EC Generic Feb 2012 Levonorgestrel 0.75 EC Generic April 2012 Levonorgestrel Implant Generic May 2010 Misoprostol Oxytocic Generic & SRA appr. May 2012 Norethisterone enantate + estradiol valerate Injectable Innovator June 2011 Oxytocin Oxytocic Generic July 2009 Oxytocin Oxytocic Generic October 2010 During the previous 12 programme months 3 new applications for prequalification were received (2 ECs, 1 Misoprostol), which all passed the screening process. Dossier review and Copenhagen assessment sessions proceeded as planned. In 6 of 7 of these assessment sessions dossiers of RH products were assessed. However, the number of dossiers received is far less than the planned number of 10. Even if all companies that participated in the second ERP in April 2012 fulfil their commitment to submit applications for prequalification for those products that are not yet in the prequalification pipeline the target might not be achieved. To further encourage submission of dossiers WHO PQP has published in 2012 an accelerated procedure for accepting submissions for RH medicines, which specifies reduced documentation requirements for stability at the time of submission.20 The impact of this accelerated procedure is not known. Ten inspections of FPP sites were foreseen until the end of 2012 with the expected result of 5-8 products’ manufacturing sites passing WHO GMP inspection (Milestone Reporting Chart). Between September 2011 and August 2012 6 RH FPP sites had been inspected of which 2 were compliant. In addition, 2 RH API manufacturers were inspected in the context of their application for 19 The 4 categories are: Oral hormonal contraceptives, injectable hormonal contraceptives, implantable contraceptives, and oxytocics. 20 WHO PQP. Accelerated procedure for accepting Reproductive Health product dossiers for assessment by the WHO Prequalification of Medicines Programme. 20 March 2012. http://apps.who.int/prequal/info_applicants/Guidelines/accelerated_procedure-RH.pdf Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 11 prequalification, and 1 API manufacturer re-inspection is planned for December (related to the applications for prequalification for two FPPs). The inspection details provided in Table 4 show that 2 RH FPP sites passed GMP while an additional 2 RH FPP sites are likely to be declared compliant without re-inspection. This result is below the target of 5-8. Underachievement is also due to the fact that the number of applications was less than planned, impacting on the number of related inspections conducted. Table 4: Overview of RH product related inspection activities and outcomes # Type of inspection Result Outlook 1 FPP site Compliant 2 FPP site Compliant 3 FPP site Non-compliant Awaits re-inspection 4 FPP site Non-compliant Awaits re-inspection 5 FPP site Awaiting CAPA Likely to be declared compliant 6 FPP site Awaiting CAPA Likely to be declared compliant 7 API site Awaiting re-inspection in December (related to 2 FPP applications) 8 CRO Compliant 9 CRO Compliant 10 API Non-compliant 11 API Compliant on re-inspection (same as #10) Related to prequalification of 3 APIs The first progress report for the supplement grant already states that - taking into consideration slow progress with dossier assessments due to manufacturers’ capacity constraints and slow response rates as well as progress with achieving GMP compliance - achievement of the target value of 5-8 prequalified products by end 2012 is unlikely. This is confirmed by the findings as per August 2012. The Public Assessment Reports of prequalified products posted on the WHO PQP website provide an overview of the steps and actual timeframes for completion of the product assessment, which allows assessing reasons for any delays. Assessment reports for the more recently prequalified RH products from generic manufacturers were not yet available, though. 3) Capacity building of manufacturers and regulators Indicator target 12/2012: 6 technical assistance missions carried out by Concept and WHO Planned activities related to this indicator are 3 training workshops co-organised with UNFPA and 2 training workshops co-organised with Concept Foundation (i.e. a total of 5 not 6). The 3 regional training workshops organised together with UNFPA were implemented in Ghana, Namibia and Tanzania. They consisted of a 3 day workshop on RH medicines and devices quality mainly for medicines’ regulators and staff of quality control laboratories and accreditation bureaus, and – aimed at a broader audience also including manufacturers - training on the new UNFPA quality standards. WHO PQP and Concept Foundation organised a 3-day workshop for RH product manufacturers in June 2012 in Beijing, China. Forty five technical and managerial staff from 12 companies attended.21 The second workshop of this type is planned to be held mid November 2012 in India. 21 Workshop on WHO prequalification of reproductive health medicines – Participants handbook. NPFPC, WHO, Concept Foundation. Beijing, China: June 2012 Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 12 With regard to provision of guidance documents the following could be traced on the WHO PQP website:  Frequently asked questions: prequalification of medicines for reproductive health (updated and expanded, 20 March 2012)22  Briefing note of 17 April 2012 covering technical support available; specific RH medicines quality, safety and efficacy issues; the ERP mechanisms and others (this document was only found under the category ‘A-Z listing of documents’, probably negatively impacting its visibility)  Briefing paper of 27 April 2012: Expert Review Panel: A rapid quality risk assessment mechanism for assessing needed pharmaceutical products that have not completed a stringent assessment.  Recommended comparator products: Reproductive Health medicines. Guidance Document, WHO PQP, 20 March 2012  Accelerated procedure for accepting Reproductive Health product dossiers for assessment by the WHO Prequalification of Medicines Programme. Guidance Document, WHO PQP, 20 March 2012 In addition, the draft guideline on quality risk management has been posted for comments at http://www.who.int/medicines/areas/quality_safety/quality_assurance/projects/en/index.html. The guideline was adopted with some modifications by the Expert Committee at its meeting in October 2012. 4) Quality and efficacy of technical assistance to RH medicines manufacturers There is no specific indicator of success related to this area. The Memorandum of Understanding (MoU) between WHO and Concept Foundation was signed on 18 January 2012. The roles of each party and areas of collaboration are broadly defined. Clauses on confidentiality regarding information shared for the purpose of collaboration are included. Provision for addenda providing more detail on specific areas for collaboration is made. It seems that no such addenda were deemed necessary to date. Based on the MoU there are frequent face-to-face interactions to clarify technical issues and related to approaches for technical assistance provided to manufacturers. Concept Foundation technical advisors also participated in a meeting for technical assistance providers (WHO technical standards, common problems seen related to RH manufacturers’ dossiers and GMP status). WHO still sees the opportunity to improve sharing of information on manufacturers with potential for prequalification or being supported by Concept Foundation, and to have standardised approaches to select manufacturers for technical assistance provision and monitoring their progression towards prequalification. While we could find no clear evidence for (or against) effectiveness of technical assistance, the few Concept Foundation supported manufacturers that were interviewed by the RT provided examples for how this helped them (e.g. doing GMP assessments; conducting workshops; identifying and explaining issues of non- compliance; assistance with improvement plans; solving specific technical questions related to bio- equivalence). Pre-audits of RH manufacturers’ facilities at the time of dossier submission were planned to provide opportunity to manufacturers to start addressing identified GMP issues early on and improve 22 This document was produced in collaboration with Concept Foundation Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 13 efficiency of WHO PQP formal GMP inspections. Pre-audits were planned to be contracted out to a SRA (budget assumes 5 in total). This did not happen because (i) the fees that could be offered by WHO to selected SRAs were not attractive enough, and (ii) only 2 new applications of generic manufacturers without SRA approval had been received.23 Alternative ways to implement the pre- audits are currently being considered by WHO PQP. 5) The business case There is no specific indicator of success related to this area. Using the findings of the business case work for strategy development is expected to impact achievement of the indicator targets in output area 2 (see above). The majority of stakeholders consulted (including WHO PQP and manufacturers) and relevant documents reviewed note the lack of a business case as the main reason for the relatively low interest of RH medicines manufacturers to apply for prequalification. WHO PQP had previously worked with BioBridge Strategies for development of business cases for HIV/AIDS, Malaria and TB medicines. Concept Foundation and BioBridge Strategies on request of WHO PQP conducted an initial survey of RH medicines manufacturers in early 2011 regarding their economic and business rationales for engaging to have products prequalified by WHO.24 Initially it had been planned that Concept Foundation and WHO PQP (through a sub-contract to BioBridge Strategies) would be working together on further developing the business case. For this BMGF had approved a pooling of available funds. However, at a meeting in July 2012 it transpired that the perceptions regarding the need / focus of the business case differed between WHO PQP and Concept Foundation. It was then decided that WHO would use its programme funds to contract this work out to BioBridge Strategies who will include manufacturers from additional geographic areas. First results of this work are expected by November 2012 (original completion date: Jan 2012). Concept Foundation proposed to write up a portion of the business case, with a focus on specific product types and including procurers’, countries’ and other RHSC stakeholders’ perspectives (planned to be available end 2012). 6) Rapid uptake of quality-assured RH medicines There is no specific indicator of success related to this area. The second critical milestone for Objective 1 (first accelerated approval of WHO PQ RH medicine by March 2012) has not been met but the deadline was also not linked to the related activity milestones (1.13). The updated milestone chart (August 2012) foresees completion by December 2012 but anticipates delay. The procedure for accelerated registration of WHO prequalified products was approved by the Expert Committee in October 2012. The procedure allows for information sharing between NMRAs and WHO PQP of detailed assessment and inspection reports with prior approval by the manufacturer and under specific confidentiality agreements and procedures. Participating NMRAs commit to process related applications for registration within a period of 90 days. Ten of the countries interested in participating in the pilot signed the confidentiality agreements. From these, four countries were of interest to the two generic manufacturers with prequalified products to register the medicine in these countries’ markets. Even if practical work starts immediately after approval of the procedure in October 2012 it is highly unlikely (given the 90 days processing time) that the first accelerated product approval will be achieved by end 2012. 23 WHO PQP requested BMGF to redirect the related funds to assist the Sri Lankan NMRA with a study on how serious quality defects detected in a RH product could be avoided in the future, but this request was not approved. 24 Concept Foundation. Survey of Reproductive Health manufacturers – Building the Business Case for the World Health Organisations Prequalification Programme (WHO-PWP). Final Draft May 2011 Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 14 7) Interim approval of RH products Indicator target 12/2012: 10 expert review panel (ERP) dossiers assessed This activity area relates to organisation of assessment of submissions and preparation of related risk rating reports by WHO PQP. Funds to conduct technical assessments of dossiers submitted by RH manufacturers to UNFPA under the ERP procedure are provided to UNFPA through Concept Foundation under the QuRHM programme. For this WHO PQP is being paid at a fixed rate per dossier assessed. For the first ERP for RH medicines (Oct 2011) only 4 submissions were received, of which only one was eligible. Eligibility was therefore amended for the second ERP (April 2012). 33 submissions were received by UNFPA and after initial screening 23 dossiers went into full review by WHO PQP. By mid-August 2012, 8 products had received a positive risk rating (Category 1 or 2); while for the remaining 15 additional information had been requested.25 4.2.3 Financial execution WHO PQP provided an updated budget report template documenting expenditure up to 30 June 2012 (9 programme months). This was used to assess financial performance in terms of % of budget spent per category (Table 5). By end June 2012 the only category overspent (as per 9 months budget estimate) was “Direct FTE” for Objective 1 (51%) leading to an overall relative overspent for Total Direct FTE costs of 33%. This was compensated by under spending in all other categories. For example, nothing had been spent on Objective 2, and the direct travel budget was under spent by 54%. In terms of overall budget estimates 80% of the total amount estimated for the first 9 months was spent, and 48% of the overall programme budget. Table 5: Financial status BMGF funding of WHO PQP RH as per 30 June 2012 total budget 15 months average monthly budget expenditure @ 3 months expenditure @ 9 months % spent of 9 months budget % spent of total budget @ 9 months Total Direct FTE 558,693 37,246 158,600 445,677 133 80 Objective 1 453,405 30,227 133,800 411,884 151 91 Project Management 105,289 7,019 24,800 33,793 53 32 - Total Direct Travel 333,996 22,266 53,731 108,087 54 32 Objective 1 333,264 22,218 53,731 108,087 54 32 Objective 2 732 49 0 0 0.00 0.00 - Total Direct Consulting 188,850 12,590 25,522 56,716 50 30 Objective 1 188,850 12,590 25,522 56,716 50 30 - Total Direct Supplies 95,000 6,333 0 15,918 28 17 Objective 1 95,000 6,333 0 15,918 28 17 - Subtotal MDC 1,176,539 78,436 237,853 627,298 89 53 Indirect costs MDC 176,481 11,765 35,678 94,095 89 53 Grand subtotal MDC 1,353,020 90,201 273,531 721,393 89 53 - Total sub contracts 115,000 7,667 0 0 0.00 0.00 Objective 1 30,000 2,000 0 0 0.00 0.00 25 The total number of products with a positive risk rating increased to 12 by 2 October 2012. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 15 total budget 15 months average monthly budget expenditure @ 3 months expenditure @ 9 months % spent of 9 months budget % spent of total budget @ 9 months Objective 2 85,000 5,667 0 0 0.00 0.00 - Total sub grants 8,340 556 0 0 0.00 0.00 Objective 1 8,340 556 0 0 0.00 0.00 - Total Sub grants/contracts 123,340 8,223 0 0 0.00 0.00 Indirect cost 18,501 1,233 0 0 0.00 0.00 - Total direct costs 1,299,879 86,659 237,853 627,298 80 48 Total indirect costs 194,982 12,999 35,678 94,095 80 48 Grand total costs 1,494,861 99,657 273,531 721,393 80 48 - Total direct by objective 1,299,880 86,659 237,853 627,298 80 48 Objective 1 1,108,859 73,924 213,053 593,505 89 54 Objective 2 85,732 5,715 0 0 0.00 0.00 Project management 105,289 7,019 24,800 33,793 53 32 As per grant agreement, variations between budgeted and spent amounts of more than 10% need to be explained. Regarding the over expenditure on FTE direct costs we were told that WHO/QSM committed the largest part of the WHO PQP/RH FTE budget at the start of 2012. This was done for practical reasons, as it is very cumbersome to make funds for monthly salary payments available by apportioning the exact amounts of staff time spent to the specific work areas supported by the different donors. If at the end of the period it turns out that the originally budgeted staff time was not according to the actual workload related to the programme, committed fund will be booked back to the respective grantee’s account.26 For WHO PQP/RH this will happen for FTE estimates related to dossier assessment, because much less dossiers than planned were received and consequently less FTE equivalents were needed for assessments. The largest part of the budget for Objective 2 was allocated to contracting the development of the business case and pre-audits. There is no specific staff time allocated to Objective 2 (e.g. for meetings to be held with Concept Foundation), and we assume that this is covered under the FTE budgets for Objective 1. Pre-audits were not implemented (see above), while work on the business case was contracted only after 30 June 2012. It should be noted that while pre-audits feature under Objective 2 in the budget, this activity is included under Objective 1 in the Narrative Supplement Request and the Milestone Reporting Chart. Overall it can be expected that the full grant amount will not have been spent at the end of the supplement grant period, which is in line with the reported progress on milestones. Taking as a reference the spending rate as per 30 June 2012 only 60% of the total grant might have been used by end 2012. 26 This is provided for in the WHO Financial procedure guideline FIN PRO 10-002, Version 1.0 – August 2010. The same guideline establishes responsibilities and authority of the Award Manager, and provides procedures for financial management and reporting of grants earmarked to projects within WHO. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 16 4.2.4 Business processes of the WHO Prequalification Programme for Medicines The process steps involved in prequalification of medicines are summarised in Annex 5. In line with good regulatory practice WHO PQP established key performance indicators and targets for the programme covering many of these process steps. Some specific indicators and targets were agreed with the relevant donors (UNITAID and later also BMGF). Selected indicator values and targets that provide information on timelines for prequalification and that were reported to UNITAID in 2010, 2011 and 2012 are documented in Table 6. Table 6: Selected WHO PQP key performance indicators (actual values and targets) # Indicator description 2010 2011 2012 (Jan-Jun) Target 2012 1 Total # of days taken to prequalify medicines (median [M], range [R]) M: 663 R: 22-2903 M:794 R: 113-1589 N/A < 547 (Target 2011) 2 Median # of days from receipt of application for prequalification to completion of initial screening of the dossier M: 7 R: 0-48 M: 11.5 R: 0-95 M: 7.5 R: 0-42 < 20 3 Median # of days from completion of screening and acceptance of dossier for assessment to completion of dossier assessment, minus "stop clock" time M: 276 R: 15-914 M: 267 R: 44-548 M: 347 R: 181-502 < 270 4 Median # of stop clock days from completion of screening and acceptance of dossier for assessment to completion of dossier assessment M: 414 R: 0-2394 M: 514 R: 2-1187 M: 385 R: 63-948 < 450 5 # of inspections performed 59 90 48 (18 FPP, 19 API, 6 CRO, 5 QCL) 75 6 Median # of days from the date of acceptance of an application for assessment, or if the manufacturer is not ready when the application is accepted, from the "ready date" provided by the manufacturer, to conduct an initial inspection Results reported here for FPP only. M: 200 R: 51-313 Results reported here for FPP only. M: 141 R: 22-226 Results reported here for FPP only. None scheduled < 180 7 Average # of website visits/month 4,600 5,286 N/A 3,800 (Target 2011) Regular monitoring of all WHO PQP key performance indicators has been a challenge as only semi- automated systems were used (spread sheets). As a consequence indicator values for the group of RH products alone were not readily available. A new IT system is currently at the stage of user acceptance testing and is expected to go live end 2012 / early 2013. This system will be able to easily generate updated information on performance measurements, and will allow timely identification of process inefficiencies that subsequently can be addressed. What is notable is the great range observed for some of the indicator values. This indicates that there is a great difference in the quality of specific dossiers and/or responsiveness of manufacturers. Indicator # 4 – the one with the greatest ranges - shows a large difference in time taken by manufacturers to respond to queries, which might be due to e.g. the amount of work needed to address specific questions, or the fact that management does not always prioritises prequalification related activities. It is not clear how much responsiveness would be amenable to increased efforts of follow-up by WHO PQP staff. Variations for Indicator # 3 can be affected by capacity constraints of Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 17 WHO PQP. The time needed to complete dossier assessment also depends on the number of (repeat) assessments required after the request for additional comments, i.e. on the quality of the submission. The fact that the overall target was not achieved allows the conclusion that there is room for efficiency improvements. The centralised registration procedure for generics at the European Medicines Agency establishes a benchmark for assessment time of 210 days (against 270 for WHO PQP), excluding stop-clock days and allows maximum 60 days for applicants to provide written responses. Direct comparison with WHO PQP timelines is however not very valid because of differences in organisational set-up, mandates, and capacity of applicants. Similar indicators related to assessment of dossiers under the ERP process are not being monitored. The responsibility for this would lie with UNFPA on whose behalf ERPs for RH products are being conducted. The following timelines for processing through the ERP mechanism can be established for the second ERP for RH medicines:  Receipt of dossiers by UNFPA: 4 June 2012  Receipt of dossiers by WHO/QSM after initial screening by UNFPA: 11 June 2012  Receipt of an additional 5 dossiers by WHO/QSM: 26 June 2012  Decisions on positive risk rating on 29 June, 10 August, and 2 October 2012 The time from the ERP call for expressions of interest closure date to decision on risk category thus varied between 26 and 95 days, the latter for products where additional information was requested from the manufacturer. These findings indicate efficient management of the process by both UNFPA and WHO/QSM. Before and during assessment of submissions for prequalification WHO/PQP provides for engagement with manufacturers. The possibility and related procedure to meet and/or communicate through teleconference or e-mail is published on the WHO PQP website. However, so far manufacturers of RH products did not use these opportunities as much as manufacturers of other product categories for which expressions of interest for prequalification are published. We were told that during the period under review only a few e-mail and telephone interactions with RH product manufacturers took place, and not a single face-to-face meeting.27 One reason could be that manufacturers who are supported by Concept Foundation under the QuRHM programme discuss their issues with the technical experts of Concept Foundation who – in turn - are in regular contact with WHO PQP. Not all RH medicines and API manufacturers interviewed in the context of this review (at the September 2012 Copenhagen meeting and at the October 2012 global pharmaceutical industry conference (CPhI) in Madrid) were aware of direct or virtual meeting opportunities28. This is in line with the findings of the 2010 WHO manufacturers’ survey. Those that benefited from this type of contact all provided a very positive rating. Previous manufacturer and other stakeholder surveys noted issues related to transparency of WHO PQP’s business processes. This applied to different areas such as training and meeting possibilities not being appropriately publicised / made known; unclear procedures and requirements; interpretation of regulatory requirements; or the decision making processes. There is a wealth of 27 During a general meeting jointly organised by UNICEF and WHO/PQP for manufacturers in September 2012 (out of the review period) 3 RH product/API manufacturers had 1 to 1 meetings with WHO PQP technical staff (out of 28 total meetings held). 28 Due to the limited number of relevant interviewees these findings are indicative and not quantifiable. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 18 information published on the WHO PQP website. This includes formal guidance on requirements and procedures; public assessment and inspection reports; status of products in the prequalification pipeline; training material; and presentations delivered at stakeholder workshops during the past years that provide detailed information and analysis of WHO PQP procedures, activities and outputs. Especially informative are the public assessment and inspection reports where in addition to detailed technical information, actual timelines related to the prequalification process of a specific product are summarised. However, some time needs to be dedicated to first identify and then process the information provided. This might be especially challenging for non-English speaking visitors to the site (Chinese manufacturers tend to mention this as a problem)29. Consequently some of the transparency issues could be related to communication and presentation. Simply making information available in the public domain or at stakeholder meetings might not be sufficient to address the concerns of all stakeholders. The procedure for prequalification30 states that a standard operating procedure exists for handling disagreements that may arise between WHO PQP and a manufacturer. However, we could not trace this disagreement procedure on the WHO PQP website. Lack of an adequate issue resolution process was also mentioned by participants of the 2010 WHO manufacturers survey. WHO PQP reports that an explanation for how to proceed in case of disagreement was added to the inspection and assessment report templates and that disagreements and responses are being logged. So far very few manufacturers were reported to having used this procedure. A frequent complaint of manufacturers and some procurers is that WHO Standards (or their interpretation) are more stringent than those of other SRAs. Often specific examples are not provided; those mentioned were related to number of months testing for stability data, and number of batches for validation.31 While WHO and SRAs rely more or less on the same international standards these are open to interpretation to a certain extent. WHO GMP guidelines tend to provide more detail on how standards can be met, and if inspectors see this as the only way it might be perceived as having ‘more stringent’ requirements. It is not easy to draw the line between acceptable risk and potential threat to public health and at the same time being concerned about availability of quality products to the final beneficiaries. In the field of contraceptives this is even more difficult, because products are taken by (normally) healthy individuals and there is still suspicion towards these products in many cultures – putting an even greater weight on any quality issues occurring in the field. On the other hand, there are quite a number of hormonal contraceptives marketed in less developed countries for many years without being quality assured according to SRA standards. Comprehensive evidence for or against their safety and efficacy and related impact on public health is not available. There might be need for the WHO Expert Committee on Specifications for Pharmaceutical Preparations as the standard setting body to refocus its approach towards minimum standards required to protect public health rather than towards current best practice. 4.2.5 Project management The position of programme manager for the overall WHO PQP has been vacant since the start of the supplement grant,32 the main reason being the currently very restrictive recruitment rules of WHO which provide a great disincentive to prospective candidates. This vacancy obviously affected overall 29 WHO PQP now has access to Chinese and Russian speaking inspectors. Between 2006 and 2010 the majority of technical assistance organised by WHO PQP was for Chinese manufacturers most of it addressing GMP. Still language barriers persist as pointed out during presentations at the CPhI in Madrid and the manufacturers meeting in Copenhagen. 30 WHO Technical Report Series, No. 961, Annex 10. WHO Geneva: 2011 31 Examples come from the 2010 manufacturer survey done by WHO PQP / Interclarity Research and discussed at the WHO PQP stakeholder workshop 2011. The relevant presentation foresaw that recommendations from the workshop would be forwarded through the Secretariat to the Expert Committee. One concrete action was reducing the required number of batches http://apps.who.int/prequal/trainingresources/pq_pres/stakeholders_2011/presentations/Day_2/WHO_GMP.pdf 32 A new programme manager was appointed in the meantime and started work end of September 2012. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 19 programme direction, supervision, and management. Many of the programme manager’s tasks had to be performed by staff members of WHO/QSM and PQP who were already stretched to capacity. Some specific examples for affected management areas that were given by interviewees from WHO PQP are:  Overall marketing of WHO PQP also at international level  Comprehensive work planning and coherence of WHO PQP  Implementation of the business plan  Financial management and reporting Work plans are available for the different WHO PQP teams (e.g. inspection, assessment) and budgets are developed based on these work plans. Under all Phase II grants WHO PQP has been reporting timely and as required by BMGF (narrative reports, and updated budget and milestone templates). Deviations of more than 10% in the budget templates were explained to the satisfaction of the funder. 4.2.6 Outcomes, challenges and lessons learnt The outcome of the WHO PQP/RH so far is not meeting expectations, although progress towards the end of the Phase II and supplement grants was more promising. During the supplement grant period 3 more products were prequalified: 2 generic COC and one innovator injectable. However, it is highly unlikely that the current 11 prequalified contraceptive products will lead to increased availability and choice in low- and middle-income countries, as stipulated in the programme goal: those listed have been provided through family programmes for a while; the inclusion of two generic COCs did not (yet) stimulate price reductions, and the number of implants and long-term injectables remain limited which continues to affect supply security. It is also not expected that the planned number of 5-8 prequalified generic (low-cost) products will be achieved by end 2012, and important gaps remain for long-term injectables and implants. However, the number of applications received and of products that obtained a positive risk rating under the second ERP for RH medicines gives reason for careful optimism. The competitive global market for quality assured contraceptive and other RH medicines that would lead to lower prices and more supply security does not yet exist. Also, there is no guarantee that manufacturers of prequalified products will proceed with registration in all priority countries limiting impact on local level markets. From the perspective of WHO PQP the biggest challenges remain (i) the small number of applications for prequalification received, and (ii) deficiencies in dossiers and GMP standards of the received applications - the QuRHM programme is supposed to assist addressing these challenges (see Chapter 5). To encourage application for prequalification for RH products WHO PQP waived some technical preconditions for successful completion of the initial screening process for these products. It is too early to know whether this measure was effective. Additional challenges for WHO PQP relate to the difficulty to recruit adequately qualified staff due to restrictions of the WHO recruitment regulations. This places pressure on existing staff that have to cover for unfilled positions. As a result there is less time available for addressing priority areas, such as clear and transparent communication with manufacturers and other stakeholders. Dependency on donor funding and the related insecurities impact negatively on staff morale. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 20 Finally, the absence of evidence based studies on the impact of substandard quality RH products on public health makes promotion of the importance of quality assured RH products to the wider community difficult. Sustainability of the WHO PQP overall has been on the agenda for a while. The Procedure for prequalification of pharmaceutical products states that “WHO reserves the right to charge for this procedure on a cost-recovery basis”.33 The 2009-2013 Business Plan34 suggests considering a mixed funding system. In that context a proposal was/is being developed looking at the possibility for WHO PQP to charge service fees. Waivers or exemptions for ‘poor’ manufacturers would be one component of this financing mechanism. The WHO/QSM team has also just published a concept paper on long-term perspectives for prequalification including the need to build networks amongst NMRAs, and to transfer prequalification activities from WHO Geneva to these networks organised at e.g. regional level.35 Despite these activities, operation of WHO PQP overall and PQP/RH in particular will require sustained donor support in the short to medium term to be able to continue operations. 4.2.7 Conclusions and specific recommendations A comprehensive assessment of the RH product market and its characteristics was apparently not done before embarking on prequalification of RH medicines. Expectations for the programme were thus overambitious probably based on successes of PQP of HIV/AIDS medicines (prequalification of TB and Malaria medicines was less successful, especially in the beginning). It is now understood that generic RH product manufacturers in low- and middle-income countries need more time to implement quality improvements and that incentives for doing so are not as obvious as anticipated. WHO PQP is trying to address concerns raised by its clients and stakeholders regarding transparency and communication. Language will remain a barrier in the medium-term, especially for Chinese manufacturers and related to technical documents available in English only. WHO PQP is also cooperating with SRAs discussing existing standards and their interpretation. Addressing some of the immediate concerns we would like to recommend to WHO PQP the following: 1. Subsequent to the market research being carried out by BioBridge, ensure that the WHO PQ activities and resources are spent in alignment with leveraging the greatest untapped supply potential towards the programme’s goal. This could imply e.g. less active engagement with SRA approved manufacturers (perhaps charging them a fee), less workshops and activities being conducted generally for manufacturers and more directed workshops and information provision with those firms who can be game changers. This may include a reassessment of the possibility to implement pre-application processes and/or other pre-application assessments for manufacturers without previous prequalification experience (or planned pre-assessment by Concept Foundation). 2. Improve the evidence base for, and better promote the public health value of, WHO PQP to donors and procurers within the RH sector 3. Request additional funding for contracting professional support to identify whether and how the WHO PQP website can be made more easy to navigate and for implementation of the resulting redesign 33 WHO Technical Report Series, No. 961, Annex 10. WHO Geneva: 2011 34 Price Waterhouse Coopers: Business Plan for WHO Prequalification of Medicines Programme 2009-2013. 7 August 2009 35 WHO QSM: Regulator prequalification of medicines: a future concept for networking. In: WHO Drug Information Vol. 26, No. 3. WHO Geneva: 2012; www.who.int/medicines/publications/druginformation/issues/26-3.pdf Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 21 4. Develop a communication strategy to ensure that potentially interested manufacturers are pro-actively made aware of the programme and of the support that WHO PQP can provide 5. Post the procedure for dispute resolution on the WHO PQP website 6. Post a summary on complaints received and outcome of resolution processes on the WHO PQP website 7. Post annual performance reports on selected KPIs on the WHO PQP website In case BMGF decides to provide new longer-term funding to WHO PQP our general recommendation would be to review the existing project framework table and develop any new programme using indicators and milestones that are coherent, relevant and measurable. Finally, the existing donors supporting WHO PQP might want to together consider their views on the harmonisation and alignment agenda. For WHO PQP, options for establishing a multi-donor basket fund could be discussed and how this might improve efficiency and value for money for all partners. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 22 5 QUALITY OF REPRODUCTIVE HEALTH MEDICINES PROGRAMME (QURHM) 5.1 Introduction Figure 1 gives a historic overview what happened in RHSC, Concept Foundation and WHO on the issue of RH medicines’ quality before the QuRHM project.36 Figure 1: Time line of RHSC quality activities, 2005-2011 MDA: RHSC Market Development Approaches working group AQAS: Accessing Quality Assured RH Supplies programme After being introduced to RHSC in Seattle 2005, Peter Hall of Concept Foundation performed a qualitative and quantitative study on the capacity of generic manufacturers to produce RH medicines under GMP conditions. He presented the results at the Bonn 2006 Conference, where RHSC members discussed the issue of quality RH medicines: Of the 44 companies from 15 countries, less than 30% would meet the cGMP requirements of WHO, the Pharmaceutical Inspection Cooperation Scheme (PIC/S) or any stringent regulatory authority (SRA); a further 20% could comply with investment and improvements in quality management. Few companies had developed adequate registration dossiers for hormonal contraceptives.37 36 Figure presented by Concept Foundation at the TAC meeting, November 2011. 37 Hall P, Oehler J et al. A study of the capability of manufacturers of generic hormonal contraceptives in lower- and middle-income countries. Contraception 75 (2007) 311 – 317. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 23 Hall had earlier suggested developing a network of existing generic pharmaceutical manufactures in lower and middle income countries that could supply their products to people in the developing world provided that those products are of appropriate quality and are affordable and accessible. 38 At the Bonn meeting, RHSC adopted a recommendation on the principle to procure only products that have either been prequalified by WHO or approved by a stringent regulatory drug agency. 5.1.1 AQAS project 2009-2011 Concept Foundation applied as member of the MDA working group in RHSC for a grant from the RHSC Innovation Fund in 2009. A grant of USD 198,063 was approved. The Accessing Quality Assured Supplies (AQAS) Initiative was undertaken from December 2009 - February 2011, and had 3 main objectives: (1) generate a list of “good” manufacturers; (2) Support and inform potential candidates how to get prequalified; and (3) Disseminate information to manufacturers and within RHSC. The project confirmed that there was a critical mass of about 30 generic hormonal contraceptive manufacturing candidates with the potential capacity to achieve PQ within 2-3 years. Concept Foundation provided initial TA to 12 companies. Results of this support were not given, apart from number of RH medicines “in the pipeline”. The project did not have a logframe but listed 15 milestones spread over the 3 activity areas. Monitoring was done by an Advisory Committee (AC), chaired by the MDA WG leader. The AC met 4 times physically and 5 times through teleconference. Concept Foundation delivered an End-of- Project report. The Review Team (RT) did not find an external evaluation. 5.1.2 QuRHM project 2011-2014 At the request of DFID following the Kampala RHSC meeting, a sub-group of the MDA led by Concept Foundation was requested to develop an outline proposal for a one year hormonal contraceptives only intervention drawing on the AQAS experience. A meeting was organised by Concept Foundation in Geneva that included WHO, GFATM representatives and the MDA WG chair. Concept Foundation submitted the first draft in July 2011 and following consultation with DFID and subsequent revisions to extend the scope of the proposal (3 years and including drugs for maternal health) a final draft was submitted by Concept Foundation on behalf of the RHSC to DFID on 25 December. The proposal included a Business Case (including intervention summary) and Logframe for a 3 year, GBP3.5m project, running from 1 April 2011 until 30 March 2014, with the project title: “Improving market efficiencies and Value for Money of quality contraceptives and other reproductive health medicines through market entry of approved generic suppliers”. The project later became better known as the “Quality Reproductive Health Medicines” (QuRHM) project. 39 The QuRHM project aimed to achieve an orderly market transition (introduction of generic RH medicines manufacturers in the international institutional buyers’ market) over the next 5 years through: 1. Presence of a competitive roster of Quality Assured (QA) suppliers across RH product groups 38 Hall PE. What has been achieved, what have been the constraints and what are the future priorities for pharmaceutical product-related R&D to the reproductive health needs of developing countries? Commission on Intellectual Property Rights, Innovation and Public Health, World Health Organization, Geneva, 2005. www.who.int/intellectualproperty/studies/reproductive_health/en/index.html 39 The QuRHM project executed by RHSC/CF should not be mixed up with the QMRH project executed by WHO PQP Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 24 2. Uptake of QA vendors by international procurement agencies and national entities (in pilot countries) 3. Consensus and harmonization of QA definitions, policies and approaches among procurers at international level 4. Adoption of common QA definitions, policies and approaches at country level (4 pilot countries) 5. Strengthened National Medicines Regulatory Authorities (NMRAs) in pilot countries to achieve a faster, consistent approach to registration of products, and 6. Increased awareness of RH donors and less developed countries’ Governments on Quality Assurance issues to inform and facilitate decision-making. The Theory of Change was that a wider availability of quality assured affordable RH medicines that are demonstrably safe and efficacious, will lead to → Reductions in the cost of essential RH medicines through more competition → more disposable money → enabling higher volumes of product by procurement agencies and countries → providing a better method mix for women and partners → contributing to efforts to address unmet need among women and girls → reduce unintended pregnancies and maternal mortality. The proposed project goal of QuRHM was: “Improved reproductive health in developing countries particularly for poorer women and adolescent girls”. This fits the international attention for this problem, but given the many different RH interventions at country level, it will be extremely difficult (and costly due to surveys) to provide evidence attributing impact (if any) at goal level to this programme. At purpose level (“Improving reproductive health commodity security in less developed countries”) 3 indicators were proposed that measure the anticipated cost reductions, and higher volumes procured both at global and (pilot) country levels. It makes sense to use these as intermediate outcomes of the project, and the country work should focus on measuring these as a baseline for the 2014/15 impact evaluation. The logframe lists an important assumption at the purpose level: “This programme, focused on the supply side, but not on the demand side interventions, is unlikely to deliver changes to CPR and unintended pregnancy alone.” This very true statement is then immediately followed by a mitigating remark: “However, significant work is done by DFID and other donors to improve access to and quality of family planning services – of which this work is a part.” The RT has unfortunately not seen much interaction of Concept Foundation or RHSC with these other donor programmes except BMGF. This underlines the need for Concept Foundation (with/through RHSC) to seek and build stronger links with projects (e.g. SMOs, NGOs) working on the demand side barriers at country level. The priority supply side “push” focus also neglects a truth in market shaping – that a key way to influence the supply side is through influencing the demand side, i.e. creating a “pull” market or making the market less opaque to suppliers. The project was set up in 4 major output areas: 1. To substantially increase the availability of affordable quality assured RH medicines for supply to less developed countries 2. Major international procurement organisations have harmonized quality assurance definitions, policies and practices. 3. Four selected countries have adopted internationally accepted quality assurance definitions, policies and improved national procurement and regulatory criteria 4. Awareness of the QuRHM strategy and related quality issues raised among donors and other high-level stakeholders within the RHSC membership. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 25 The 4 output areas have reasonably clear indicators and milestones, except maybe those for output 4 which are not SMART enough for measuring results (See table 7 and the DFID Annual Review template in annex 11). The contract was routed through PATH (as legal body behind the RHSC) to subcontract Concept Foundation. PATH takes responsibility for financial matters, and refers questions on content matters to Concept Foundation. DFID’s original intention (stated in the DFID Business Case) was that RHSC was to be responsible for QuRHM implementation. This can be seen in the scoring in the business case option appraisal of critical success criteria: DFID gave the highest score to contracting RHSC rather than Concept Foundation directly. DFID clearly valued that RHSC would be active in supervising the Concept Foundation subcontract although Concept Foundation do not appear to have been made aware of this. (See TAC section 5.8 below). DFID has meanwhile clarified this position: the QuRHM Programme would benefit from RHSC’s strength as neutral convener and multi-stakeholder membership body, and as part of a broader basket of RHSC support DFID decided to route the project through the RHSC. The Value for Money (VfM) aspects of QuRHM were primarily defined as expected cost savings, although alternative gains (more quality RH medicines for the same money) were also mentioned. The RT noted that the business case used an assumed generic DMPA price of $0.70 in the VfM text, but the calculations are done with a lower $0.50 figure. The timing of results was in the opinion of the RT very ambitious: seeing results from the prequalification of generic RH manufacturers at country level will take much longer than anticipated particularly as the programme is not designed to have a substantial market shaping role on the demand side (see national procurement in section 7 below). 5.2 Progress on outputs first 12 months Table 7: Planned milestones & outputs vs. achievements per 31 July 2012 (12m QuRHM) Nr Planned outputs & milestones Achievements as per 31 July 2012 RT Comments (sub) score 1 Substantially increase the availability of affordable quality assured RH medicines for supply to less developed countries Steady progress, would have been A if in time A 1.1 Additional FPP and API studies completed 23 FPP (16 with ongoing technical support) and 9 API manufacturers included in the programme A+ 1.2 3 generics prequalified, 6 under ERP 2 generics Prequalified, 5 under ERP 5 more ERP recommended 10 August, and another 2 on 2 nd October A 1.3 2 RH APIs prequalified No RH APIs prequalified 4 different APIs currently under assessment B 1.4 Business case for manufacturers completed/disseminated Data collection underway, discussions held with management of companies. C 2 Major international procurement organisations have harmonized quality assurance definitions, policies and practices On track A 2.1 New UNFPA QA policy published and disseminated to all procurers QA policy available and disseminated UNFPA policy was approved pre-QuRHM! A Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 26 Nr Planned outputs & milestones Achievements as per 31 July 2012 RT Comments (sub) score 2.2 Two high-level procurement workshops One held (New York, April 2012) 2nd workshop postponed from Dec 2012 to end April 2013 B 2.3 Procurement guideline development under way No progress Re-planned for FY 2013/14 B 2.4 ERP and interim mechanisms established and functioning 5 ERP recommended in 2 sessions; no other interim mechanism 7 additional ERP recommended by 2 Oct; interim mechanisms not needed A+ 3 4 selected countries have adopted internationally accepted quality assurance definitions, policies and improved national procurement and regulatory criteria Seriously delayed Increased risk rating from Medium to High C 3.1 Completion of situation analysis in 4 countries to provide baseline No progress RFP for situation analysis published 24 Oct C 3.2 Market dynamics data in 4 countries available No progress Reports due 15 March 2013 C 4 Awareness of the QuRHM strategy and related quality issues raised among donors and other high-level stakeholders within RHSC A 4.1 Biannual reports presented to the RHSC’s Market Development Approaches and Systems Strengthening Working Groups Biannual reports presented to both working groups A 4.2 Planning for workshop at the 2012 RHSC Membership Meeting completed Concept Foundation planned for a procurers wshop, and participated in a plenary session Planned procurers wshop did not take place B 4.3 High level meetings with procurement agencies completed meetings held with 4 out of 7 identified target international procurement agencies A Although 31 July was the formal date for the Annual Review to judge progress, the RT used its discretion to also take into account ongoing processes that delivered just after the cut-off date. The (sub)scores for the outputs & milestones are identical to those used in the DFID Annual Review template (see table 2). Table 8: DFID output ratings Description Scale Outputs substantially exceeded expectation A++ Outputs moderately exceeded expectation A+ Outputs met expectation A Outputs moderately did not meet expectation B Outputs substantially did not meet expectation C For details of the progress assessment see the DFID Annual Review Template in Annex 11. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 27 5.3 Financial expenditure vs. budget A 3-year budget was allocated to QuRHM: Table 9: QuRHM Programme budget Budget Actual Indicative Item* Year 1 Year 2 Year 3 Total Output 1 452.170 452.170 430.000 1.334.341 Output 2 262.147 215.000 180.000 657.147 Output 3 80.000 120.000 165.000 365.000 Output 4 63.541 70.000 70.000 203.541 Monitoring 49.664 49.664 49.664 148.991 Total direct costs 907.522 906.834 894.664 2.709.020 Administration 90.752 90.683 89.466 270.902 RHSC Total 998.274 997.517 98.413 2.979.922 DFID/BMGF Evaluation 100.000 100.000 100.000 300.000 TOTAL 1.098.274 1.097.517 1.084.130 3.279.922 The project formally started on 1 August 2011, but pre-contract activities happened from April 2011. Concept Foundation invoices monthly in arrears in USD to PATH, which converts the amounts to British pounds (GBP) and submits them quarterly to DFID. In order to assess the spending rates, the RT compared the Concept Foundation invoiced amounts to the official project budget in the Business Case per output and financial year. For FY1: Figure 2: Concept Foundation expenditures vs. budget FY1 (GBP) Note that the DFID budget lines of “Monitoring” and “Administration” were regrouped as “Admin & TAC” in the Concept Foundation accounts. Under “Admin” Concept Foundation invoices all direct costs for the QuRHM project not covered under outputs 1-4. The Concept Foundation 10% overhead charges on indirect costs are already included in the respective outputs. The budget for the external (impact) evaluation is not yet included as that contract has not yet been concluded. Expenditures Concept Foundation (QuRHM) from project start up to 31 March 2012 (GBP actual) FY1 budget (GBP) Spending rate OUTPUT 1 - Working with manufacturers 348.890£ 452.170£ 77% OUTPUT 2 - Procurement harmonization 215.442£ 262.147£ 82% OUTPUT 3 - Country activities -£ 80.000£ 0% OUTPUT 4 - Advocacy and communication 27.317£ 63.541£ 43% Admin & TAC (DFID: Monitoring + administration) 66.199£ 140.416£ 47% Subtotal CF invoices to PATH 657.848£ 998.274£ 66% PATH costs 23.337£ -£ Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 28 For FY2 the RT received data for the first 6 months: 1 April – 30 September. These are compared to 50% of the FY2 budget. Please note that the PATH costs only cover one quarter (1 April – 30 June 2012). Figure 3: Concept Foundation expenditures vs. budget first half FY2 (GBP) The low spending rate in FY1 (68%) was partially compensated by picked up spending in first half of FY2 (110%). The total picture from project start until September 2012 now looks as follows: Figure 4: Concept Foundation expenditures vs. budget from project start up to 30 Sept 2012 (GBP) Overall spending rate is still below expectation at 80%. PATH gave the following reasons for the underspent to DFID: (1) the long negotiation process between Concept Foundation and UNFPA for output 2; (2) the forecasting assumed a linear burn rate which is not entirely accurate for actual spending; (3) country level work (output 3) will be picking up in November 2012. The final list of pilot countries was only decided in August 2012; (4) Output 1 will also be seeing an increase in personnel spending because recruitment of planned additional technical resources after September 2012; and (5) slow start of the project due to delay in getting an agreement between PATH and DFID. The RT agrees with the underspent explanation, but still believes that Concept Foundation could have started earlier preparing for output 3 and increasing activities in output 4. The apparent overspend by UNFPA in output 2 is caused by advance payments at specific dates as per the recently revised Concept Foundation-UNFPA contract. Due to the late signing, UNFPA has not yet given its 1st financial report to Concept Foundation, so the output 2 amounts may not (yet) represent real expenditures. Expenditures Concept Foundation (QuRHM) FY2Q1+Q2 April- Sept 2012 (GBP actual + estimate) 50% of BC FY2 budget Spending rate OUTPUT 1 - Working with manufacturers 265.255£ 226.085£ 117% OUTPUT 2 - Procurement harmonization 219.916£ 107.500£ 205% OUTPUT 3 - Country activities 5.328£ 60.000£ 9% OUTPUT 4 - Advocacy and communication 8.686£ 35.000£ 25% Admin & TAC (DFID: Monitoring + administration) 45.280£ 70.174£ 65% Subtotal CF invoices to PATH 544.465£ 498.759£ 109% PATH costs 2.611£ -£ Expenditures Concept Foundation (QuRHM) from project start up to 30 Sept 2012 (FY1+ 50% FY2; GBP actual + estimate) FY1 + 50% FY2 budget Spending rate OUTPUT 1 - Working with manufacturers 614.145£ 678.255£ 91% OUTPUT 2 - Procurement harmonization 435.359£ 369.647£ 118% OUTPUT 3 - Country activities 5.328£ 140.000£ 4% OUTPUT 4 - Advocacy and communication 36.003£ 98.541£ 37% Admin & TAC (DFID: Monitoring + administration) 111.478£ 210.590£ 53% Subtotal CF invoices to PATH 1.202.313£ 1.497.033£ 80% PATH costs 25.948£ -£ Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 29 5.3.1 Value for Money The RT could not easily assess VfM as Concept Foundation had not been requested in the DFID contract to provide VfM reports. In the absence of VfM reports, the RT tried to assess VfM based on unit costs and output performance per GBP. Concept Foundation unit costs are mentioned in the financial annex to the pre-contract budget offer to DFID. Consultant fees are reasonably affordable at USD 500-USD 650 per day. Per diems are standard at USD 350 (USA) and USD 250 (elsewhere). The transport policy of Concept Foundation (business class for intercontinental flights > 6hours) is different from that of sponsor DFID (economy only), and may lead to higher unit costs for travel. Concept Foundation, PATH, UNFPA and WHO all follow their own organizations’ travel policy, stating there was not an additional clause in the terms of the agreement between PATH and DFID (and therefore not between PATH and Concept) requiring either organization to follow anything other than their own policies. Assessing VfM in the Concept Foundation subcontracts was based on comparing expected outputs in the budget with the total costs. WHO did show unit costs in the budget that were found to be standard. UNFPA did not yet provide financial accounts to Concept Foundation on its advance payments, and referred to the UN “Single Audit” policy when the RT requested details of staff time used for QuRHM. Due to the multi-layered subcontracting (e.g. DFID-PATH-Concept Foundation-UNFPA-WHO for the ERP work), the management overheads and/or project support costs added at each step add up to 33%, broken down as follows: 10% by Concept Foundation, 7% by UNFPA, and 13% by WHO. But this was known and accepted by DFID at the time of the project setup. The actual staff and reimbursable costs by PATH are rated as good VfM. VfM in the QuRHM Business Case is largely based on anticipated cost reductions of hormonal contraceptives when more generic products will be prequalified. The RT did not expect Concept Foundation to make an impact in FY1 or FY2. For the future, impact may require attention to specific product areas: 71% of UNFPA’s purchasing of hormonal contraceptives in 2009 was spent on 2 originator implants and 1 originator DMPA. Unfortunately no generic implant or DMPA has since been prequalified or ERP recommended, and none are in the PQ application process. The generic implant is still a few years away from being prequalified due to lengthy clinical trials. The RT noted that one of the branded implants’ manufacturers agreed to a substantial price reduction after a volume agreement brokered by CHAI. COCs make up 39% of RHSC members hormonal contraceptives procurement by value. Two COCs got prequalified, and 6 got ERP approval in 2012. The anticipated price reduction of COC in the DFID Business Case was a drop from $0.29 to $0.20 per cycle. However, performance on VfM could not be confirmed yet as (1) the two prequalified generic products have not yet been able to reach many markets, one of them because the product is not yet registered in many countries, and (2) two of the 3 products available in the UNFPA catalogue (one prequalified generic COCs and one ERP- recommended COC) are more costly than existing originator products.40 40 UNFPA Catalogue: http://www.unfpa.org/public/home/procurement/AccessRH and personal communication UNFPA staff. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 30 5.4 Efficiency of Concept Foundation processes per output area Output area 1 - How Concept Foundation identified and categorized manufacturers Concept Foundation has presented its methodology how it identifies and assesses potential RH manufacturers at various RHSC conferences, and how it refined since 2006. After an open search, own research and info from PQP, it (1) sends potential RH companies information and a letter of consent; (2) sends a cGMP technical checklist, and reviews their answers; (3) sends a detailed technical questionnaire, analyses the replies, and categorizes the company as potential RH medicines supplier under QuRHM; (4) does an on-site, 4-days assessment. Companies are grouped in 3 categories: (A) supplying to national markets, some tenders, wishing to increase exports and interested in PQ; (B) same but not interested in PQ; (C) intending to supply SRA markets. Support is primarily focused on group A. The final ranking for deciding on support is based on a ranking in 4 categories: (A) interested in prequalification and capable, with support of achieving prequalified status for one or more of their products; (B) demonstrating clear potential technical capability to achieve prequalification status for their products, but lacking to differing degrees interest in the WHO PQP at the present time; (C) possessing the technical capabilities but not representing realistic targets for short to medium term prequalification. These prospects have established a clear direction in terms of their product development, target markets and commercial aspirations and do not consider the prospect of competing for public/social marketing sector business at low unit costs as part of their immediate business model; (D) companies who lack the technical capability, interest and attitude to manufacture hormonal contraceptive products of assured quality and do not have the ability or the willingness to invest to meet the requirements of the WHO programme in the foreseeable future. As per October 2012, 9 API companies and 23 finished pharmaceutical product (FPP) manufacturers are in the QuRHM programme, covering all RH medicines except Magnesium sulphate injection. As presented elsewhere, this has so far resulted in 2 generic RH medicines being WHO prequalified and 12 passing ERP. The Concept Foundation method is judged technically sound by the RT. However, the original requirement in the DFID Business Case was not met (for RHSC to have published an open EoI to allow any interested RH manufacturer to participate). Verifying Concept Foundation’s efficiency and effectiveness how it used 51% of its overall expenditures in output 1 is hampered by the fact that Concept Foundation did not offer access to the names or files of the companies it is or has been supporting. The resulting tables in progress reports only show coded numbers rather than actual company names (see Annex 10). The RT suggests signing a confidentiality agreement with Concept Foundation for the next Annual Review in 2013. The generic manufacturers that the RT independently contacted during the CPhI in October in Madrid had no problem discussing the status of their links with Concept Foundation. Similarly, the consultant team that evaluated the Clinton Health Access Initiative (CHAI) for DFID had full access to meeting with CHAI’s industrial partners and it is commonplace for UNITAID proposal reviewers to also have full access to industrial proponent’s proprietary information (upon signing a WHO confidentiality agreement).41 41 Personal communication with Cheri Grace, external reviewer. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 31 Output area 2 - Efficiency of outsourcing In FY1 four subcontracts were planned to be awarded under the programme: i. Expert Review Panel – contracted by UNFPA to WHO, building on existing procedures and costs of ERP activities facilitated by WHO/QSM on behalf of the Global Fund. ii. Batch sampling and testing of hormonal contraceptives – part of the interim quality assurance measures, contracted by UNFPA to Family Health International with costs based upon received quotations. iii. Testing of hormonal contraceptives – as part of the manufacturer assessments, contracted by Concept Foundation to Health Concepts International (Concept Foundation’s subsidiary company) with costs based upon received quotation. iv. Situational analyses in four countries, to be contracted by Concept Foundation using an LIB tender process. Contract iv is not yet in place – the RFP for the work was issued 24 October 2012. Results are now expected 15 March 2013. Concept Foundation had problems implementing its initially signed contract with UNFPA, as senior UNFPA management found it not to be in agreement with UNFPA administrative requirements. In the protracted negotiations for a revised contract UNFPA requested budget items that were not envisaged in the QuRHM project design. Concept Foundation and UNFPA finally signed a revised contract in September 2012 that adhered to the project plan, covering the period 1 January 2012 – 31 March 2014. Under the contract UNFPA is – amongst others - responsible for organising meetings with procurers. The delayed signing did not interfere with the ERP process as WHO was prepared to start the ERP work without a contract being in place. No VfM reporting requirements were included in the UNFPA subcontract. UNFPA is being paid in advance, and provides financial reports every 6 months and an annual report. As reports have not yet been provided (sub-contract only signed in September 2012) , the RT could only compare the total budget and listed unit costs with the contracted outputs to assess VfM. The RT did not receive information from UNFPA on the exact number of FTEs to be delivered to QuRHM by the 3 listed QA Associate positions at a total annual cost of USD 243,000. The unit cost of a consultant for Guideline Development on Procurement Practises is USD 1000/day, more than the standard Concept Foundation or WHO PQP rates, but still within acceptable range for specialised consultants. WHO PQP was again subcontracted by UNFPA for the ERP process. The requested lump sum of USD 5000 per ERP dossier sounds reasonable. Output area 3 – Outsourcing country work/baseline Efficiency cannot be assessed yet as there was no activity until October 2012, when the RFP for country work was launched. The 4 pilot countries were only decided in Aug 2012 by POC. Output area 4 – Raising awareness in RHSC Raising the awareness of QuRHM programme and related quality issues among procurers has been included in the UNFPA subcontract – see under output area 2 for an assessment of efficiency. Raising awareness of QuRHM programme and related quality issues among donors and RHSC members is discussed under RHSC stakeholder management below. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 32 5.5 Concept Foundation internal business procedures Concept Foundation is a not-for-profit foundation with offices in Bangkok and Geneva. The Director of Operations (LC) is the Programme Manager. The CEO (PH), is overseeing progress and is also acting as technical resource for output 1. Technical support was coordinated by the (former) Chief Technical Officer (HZ), who is handing over his job to a new full-time Head, Technical Services (AT). The Concept Foundation management team knows each other well, and has built a strong reputation in supporting (generic) manufacturers over time. Beyond the core business of interacting with manufacturers (output 1), the organisation has complemented its own strengths using additional partners to perform all the work. The subcontract with UNFPA should take care of output 2 and (partially) 4; the planned contract with external consultants should take care of output 3. Output 4 also requires a lot of advocacy by Concept Foundation itself. Concept Foundation may not have realised the workload that comes with still being managerially and legally responsible for the outsourced deliverables. The CEO is also very much involved in the daily business as a committed technical consultant. This makes reporting lines somewhat unclear (a case in point is the draft organogram in which staff is reporting to both the CEO and Director Operations). The RT has the impression that the increased workload is taking its toll on Concept Foundation management, as several respondents remarked that e-mails frequently are answered late or even remain unanswered. Concept Foundation is addressing these challenges by now recruiting additional staff to strengthen its management and programme support teams. The RT recommends that the management team reorganises and expands itself to enable it to respond more adequately to the increased workload and demands, and to clarify the lines of command among top management. 5.6 Expert Review Panel (ERP) and other interim mechanisms The low number of prequalified generic hormonal contraceptives (2 COCs in 2011) poses challenges for procurers who want to buy good quality RH medicines but do not want to be dependent on originator products. In other disease categories (HIV/AIDS, TB, malaria, etc.) the interim mechanism “Expert Review Panel” (ERP) was successfully used to increase the number of products that can be procured in donor-funded programmes.42 To be eligible, manufacturers need to have a cGMP compliant manufacturing site, and have been accepted for assessment by WHO PQP or SRA. This latter requirement was dropped for the RH EoI of April 2012. Under output area 2 Concept Foundation outsourced the organisation of an ERP process to UNFPA who in turn subcontracted WHO, as per the logframe. This initial process also included an Internal Technical Committee (ITC) review. The results of the initial efforts by UNFPA in October 2011 to use the ERP mechanism for RH medicines were disappointing. Of the 4 submissions received only one was eligible, which was anyway about to be prequalified. Following consultation between UNFPA and Concept Foundation the ITC mechanism was dropped by UNFPA as it was felt that the programme should aim for broad buy-in from procurement agencies, something which the UNFPA ITC could not deliver. Concept Foundation’s consultation with WHO, UNFPA and programme donors as well as USAID resulted in modified and more flexible ERP criteria for RH medicines. A new EoI was issued in April 2012 which proved more successful: 33 dossiers were received by UNFPA; 8 of these were rejected on initial screening by UNFPA staff. 25 dossiers were forwarded to the WHO Expert Review Panel, covering 11 of the 17 invited medicines (1 implant, 5 injectables, and 19 tablets). 23 dossiers went into full review (one dropped out due to a GMP risk level 4, one was already SRA approved). By 42 http://apps.who.int/prequal/info_press/documents/ERP_article.pdf Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 33 2 October, 12 products had been ERP-approved (level 1 or 2): 6 COC, 4 ECs, 1 progesterone-only pill, and 1 oxytocic. ERP approvals are valid for 1 year (renewable under conditions). RH manufacturers interviewed by us that knew about the ERP mechanism were clear about the differences between ERP and WHO prequalification. WHO PQP published a review document on ERP specificities and experiences on the PQP website. Later WHO PQP provided a detailed presentation at the recent meeting of manufacturers in Copenhagen (September 2012). However, the main responsibility for publicising and explaining the purpose of the RH ERPs lies with UNFPA on whose behalf WHO coordinates the technical assessment work. The UNFPA QA policy still includes the ability to use another interim mechanism: its Internal Technical Committee can review the quality of those reproductive health medicines that are not covered by the EOIs for WHO PQ and ERP processes. However, the DFID Quality Assurance Policy for Reproductive Health Commodities of September 2012 does not mention the use of other interim mechanisms. Members of the TAC in November 2011 were also sceptical whether the ITC would be helpful for other procurers than UNFPA. The RT considers the ERP a useful additional interim mechanism as it increases the number of quality assessed RH products available to procurers and women in the short term. The initially disappointing results turned rather positive after manufacturers’ had been given the opportunity to provide clarification on specific questions. As experience with other product categories showed, not all ERP recommended products will finally be pre-qualified but the chances that they will are higher than for products that did not receive a positive ERP risk rating. However, after the2nd ERP the problem of having only one innovator supplier each for quality assured implants or injectables still existed. Generic implant and injectable manufacturers still need to invest in quality improvements to pass the ERP. Two more calls for EOIs for RH ERPs are currently planned for 2013. 5.7 Stakeholder management Relations with industry The Concept Foundation methodology how to identify and categorize potential generic RH manufacturers is described elsewhere. Stakeholder management with pharmaceutical manufacturers was hard to assess as Concept Foundation did not share its list of active or past active contacts with the RT. Upon our specific request, Concept Foundation sent us a list of all RH medicines manufacturers known to Concept Foundation, whether they were being serviced by Concept Foundation or not. The list contained 82 RH manufacturers, 41 were classified as A or A+, 5 B, 7 C and 29 as D. The classification is based on relevance for the QuRHM project. During the Annual Review the RT independently identified another 21 sources for RH medicines, but only a few of these would be interesting manufacturers for the QuRHM project. The RT therefore concludes that Concept Foundation has covered the RH medicines manufacturing landscape pretty well. The RT interviewed 21 generic and originator companies at the Global Pharmaceutical Industry meeting in Madrid 9-11 October. Thirteen of these were in the A category. Four generic companies voluntarily disclosed that they were (or had been) supported by Concept Foundation. Three of the four were satisfied with services provided; one was generally satisfied, but mentioned the challenges of long response times by Concept Foundation related to follow-up questions. From among the potentially suitable generic manufacturers, only one had never heard of Concept Foundation. Most Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 34 generic manufacturers reported that technical support of the type provided by Concept Foundation was in principle useful and welcome. Two would have appreciated more support in terms of information on international market opportunities, and one specifically mentioned support with sourcing acceptable comparators for bio-equivalence studies. All were supporting the ultimate goals of QuRHM. The RT concludes that Concept Foundation knows the landscape of RH manufacturers very well, and is capable of providing good technical support to generic manufacturers although a systematic and detailed assessment of this support could not be done as explained elsewhere. Relations with UNFPA, WHO Focal points were agreed with the other implementing partners UNFPA and WHO, and a regular communication mechanism was set up between the parties. For the TAC, see section 5.8 below; for more detail on collaboration with WHO PQP and joint monitoring of the two programmes see Chapter 6 below. Relations with RHSC The QuRHM project was specifically routed by DFID through RHSC to ensure the broad engagement of RHSC members and support to Concept Foundation. Developing/maintaining a strong relation with RHSC and its members would thus be important for Concept Foundation. Concept Foundation has been reporting to, and has been making excellent presentations about the QuRHM project to the MDA and SS working group meetings and teleconferences, and to the RHSC Executive Committee and the annual conferences. The reports and presentations are also transparently available on Concept Foundation and RHSC websites. The MDA WG is the main place of interaction between Concept Foundation and RHSC, and has a dedicated work stream 4 “Facilitating the availability of quality assured supplies”. Concept Foundation gave presentations to the MDA WG in 2011 and 2012. Members wondered what the best form of engagement would be. “We should talk about where quality fits into the Coalition. Currently it is spread between MDA and SS working groups, although the “joint” work stream isn’t as joint as we had intended. SS WG has interest in quality but the MDA WG has been engaged in more activities. Should there be a Caucus on Quality Issues? This issue could be raised during our session in Paris on collaboration across the Coalition. Also, last year the MDA was receiving many updates from Lester, but these have diminished. MDA members would like to hear more feedback from this project.”43 It is clear that any interested RHSC member could have taken note of Concept Foundation’s QuRHM activities. However, the RT still has the impression that Concept Foundation could make more use of the RHSC membership for some of its activities, especially the country work and advocacy towards procurers. In the DFID Business Case responsibility for the QuRHM project was originally planned for RHSC. Apart from the fiduciary support by PATH (legal entity behind RHSC) it is unclear how this responsibility was taken forward. Some aspects of the relation between Concept Foundation and RHSC are discussed in the next section on TAC. Relations with Procurers including SMOs Under output areas 2 and 4 Concept Foundation is expected to advocate for common quality assurance criteria with procurers. Some of the activities are outsourced to UNFPA. Procurers are 43 MDA WG minutes, September 2012. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 35 keen to buy quality RH medicines at affordable prices, and are in general supportive of the QuRHM programme, but have been disappointed with the limited number of prequalified generic RH products. Concept Foundation in collaboration with UNFPA organised a special meeting with procurers in New York, April 2012, as a follow-up of a special procurers meeting in Washington in May 2010. The objective of the meeting was to build consensus among procurers for a transition process towards a situation where RHSC procurers would no longer buy non-quality assured products. Particular attention was given to explaining the ERP process, which the meeting endorsed. Although the meeting suggested several proposed activities, the workshop report does not consistently indicate which organisation is in charge of coordinating / implementing them, neither was any process for follow-up on agreed activities established. A second meeting with procurers, scheduled for 2nd half of 2012, was postponed till April 2013. Concept Foundation also organised direct meetings with key procurement agencies: USAID and PSI (Washington, August 2011), MSI (London, April and July 2012) and UNFPA (various). A planned dedicated procurers’ session during the Paris meeting (3 October) was cancelled as a result of the limited number of key participants who would be attending the Paris meetings, in particular those from UNFPA. Procurers interviewed by the RT at the Paris meeting and in Washington declared their principle support for the QuRHM project and willingness to adhere to a common QA policy, but also expressed their frustration with the limited progress in getting generic RH products (especially implants and injectables) prequalified. Although the ERP results (12 products approved) were positive, no easy solution is in sight for getting a generic implant prequalified at short notice. One injectable remains under review of ERP-2, and two others are expected to apply to the 2013 ERPs.44 Relations with Countries Good quality, affordable RH medicines finally will need to find their way to millions of women in low- income countries. Output area 3 tries to identify and reduce barriers for generic quality assured RH medicines at country level. This area has so far not been getting much attention from Concept Foundation, who admits that working at country level is not its strength. There is thus an opportunity to involve and mobilize the rapidly increasing numbers of RHSC members at country level. The evaluation of RHSC also pointed out that “progress at country level has been the least of RHSC accomplishments”. A new strategy how RHSC and Concept Foundation can approach countries seems indicated. This would impact output area 3, which up to now reads too much as a top-down approach for mitigating the problems and barriers at country level. Learning lessons of what countries need or suggest themselves is crucial. 5.7.1 Transparency How Concept Foundation supports generic companies Although specific details on supported companies are not publicly available, the generic methodology of Concept Foundation towards generic companies has been well described in various reports, and seems transparent enough. Concept Foundation and WHO are also preparing business cases for generic companies, where Concept Foundation focusses on a business case for quality. 44 Personal communication, Lester Chinery, November 2012. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 36 Concept Foundation’s own products The Concept Foundation website45 transparently mentions two products that it has been working on, and that were licensed to pharmaceutical companies Sun Pharmaceutical industries Ltd in India (licensee for Medabon® and Cyclofem®) and P.T. Tunggal, Indonesia (licensee for Cyclofem®). DFID informed the RT that its lawyers had considered this information before signing the QuRHM contract and that a specific Conflict of Interest statement was signed by Concept Foundation, that requires clear demarcation of the business process of QuRHM and the other Concept Foundation projects. Some stakeholders have mentioned to us this possible conflict of interest of Concept Foundation, and it might be worthwhile that Concept Foundation considers actively and transparently addressing any doubts in that regard. 5.8 Technical Advisory Committee (TAC) As per RHSC technical offer and DFID business case, advice on the strategic direction of the QuRHM programme was to be provided by the “QuRHM Advisory Committee”, which would also ensure that biannual reports are presented to the RHSC’s Market Development Approaches and Systems Strengthening Working Groups (MDA WG, SS WG). The QuRHM Advisory Committee, later renamed as the “Technical Advisory Committee” (TAC), was supposed to meet twice a year. It has a budget line in the QuRHM budget, combined with Concept Foundation direct costs. The 1st TAC meeting was held 17 November 2011, and attended by 16 participants from 11 different organisations. The TAC was envisaged to have 3 main roles: (1) reporting back to RHSC; (2) ensuring linkages with the broader quality environment; and (3) providing strategic advice. Finalizing the TORs and membership criteria was postponed till the next meeting. It was decided that TAC would not be a closed group, but that members be invited from the wider RHSC community. Concept Foundation was given the task to act as secretariat for TAC, and Ben Light (UNFPA) was asked to chair. The meeting report showed fundamental and important discussions, and contains important new data. Follow-up was, however, limited. The planned half-year virtual TAC meeting did not take place, and the annual face-to-face meeting was not held as planned a month before the annual RHSC membership conference, but during this conference in October 2012 in Paris. Members noted that the busy 2012 timetable (Procurers meeting, ERP, FP Summit, etc.) prevented them to meet as TAC. The 2nd TAC meeting on 2 October 2012 was announced as part of the overall RHSC membership meeting programme, and existing TAC members were invited. Other RHSC members could express their interest to attend to the TAC secretariat. Of the 18 invited participants only 8 attended. WHO PQP was not represented, nor were any RHSC members who had not participated in the first TAC. The meeting could have been structured better: last year’s minutes or action points were not followed up, and the agenda was not strictly followed. Criteria for membership or TORs for the TAC were not presented or discussed. Still, some important fundamental points were debated, and good information was shared. Five weeks after the meeting, however, the minutes are not yet available. TAC meetings can discuss important issues, raise new points and suggest options on how the QuRHM project could develop. However, so far Concept Foundation has failed to mobilise the collective RHSC feedback to QuRHM. The TAC participants represent a small selected group of the RHSC membership. The TAC meetings could also be better organised with clear objectives, membership criteria and public agenda’s and minutes. From the other side, the RHSC membership and secretariat also need to play a more proactive role towards QuRHM and Concept Foundation. 45 http://www.conceptfoundation.org/industry.php Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 37 5.9 Perceptions and comments of stakeholders Concept Foundation’s work with manufacturers and the technical skills of its staff are highly valued by most respondents the RT talked to. Most accept the confidentiality that surrounds the work with specific manufacturers. Although many (especially procurers) are disappointed about the current outputs, most still want to give Concept Foundation the benefit of the doubt, as they accept that products are in the pipeline. More critical remarks were made about the erratic email responsiveness of Concept Foundation management, and the lack of communication as to why the support to a company is suddenly interrupted. In addition, there is some lack of clarity about Concept Foundation’s role with Medabon and Cyclofem API. One interview respondent wondered why these products have not yet been prequalified or ERP recommended.46 The RHSC members no doubt appreciate the regular reports, but feel that the work could be done more “jointly” in the MDA or a specifically created Quality WG. 5.10 Outcomes, challenges & lessons learned so far Outcomes After 1 year, and country work yet to start it is hard to envisage outcomes at this stage. Concept Foundation is making steady, albeit slower than expected, progress towards achieving its outputs in area 1. But even if the outputs (RH APIs and medicines prequalified or ERP recommended) are achieved soon, products will still need to pass through the phases of national registration, procurement and advocacy/promotion. The downstream work depends largely on the outsourced work areas 2-4 which represent a sizeable part of the total budget. Here the RT also noted some delays. The RT foresees that by 2014 there will still not yet be a high chance of measuring substantial impact. Given the current progress the RT is foreseeing the need for pushing back the impact evaluation to 2015. Sustainability Between 2005-2010 Concept Foundation has received modest and intermittent donor support for its work with RH manufacturers – around a $150,000 from ICON, UNFPA and PPD for the original study in 2005/6 and a further sum of $200,000 from the RHSC Innovation Fund in 2009/10, although only a small part of the latter funding was directly for support to manufacturers (the remainder for landscaping activities). The DFID grant has enabled it to provide services for free to all companies – generic or branded, SRA-based or developing country based. Long-term support is not envisaged; the idea is to catalyse sustainable market changes in some specific RH medicines markets. With the problems that some manufacturers face in achieving PQ status, Concept Foundation efforts should become more focused on the biggest needs, the lowest hanging fruits, and where it can add most value. Donors will likely also gradually shift their attention to the downstream problems (in countries). As the Value for Money of the current Concept Foundation model is hard to assess, Concept Foundation might need to consider developing other financing strategies to pay for its services. These could include charging for SRA-based companies incremental fees for GMP repeat inspections in developing countries; or introducing a limited number of free support days, after which co-payment is needed. 46 CF responded to the RT as follows: Medabon has SRA approval in 14 EU countries, the manufacturer has opted for this route rather than PQ. Work is ongoing to resolve issues relating to the supply of API for Cyclofem – a prerequisite to submission to the PQ programme. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 38 Analysis of mitigation of outside risks, comment on risk analysis The outside risks have not substantially changed, and the risk analysis should therefore remain HIGH. Validation of self-assessments and reports The regular Concept Foundation reports and presentations are always very informative, to the point and make an honest impression. However, the confidentiality agreements between Concept Foundation and manufacturers make it difficult for outside observers and the RT to validate the reports. 5.11 Conclusions and recommendations Concept Foundation is a technically competent organisation making steady progress towards achieving the QuRHM programme outputs. The number of prequalified APIs and FPPs is still low, but the ERP results and process are encouraging. As the pipeline still contains many interesting products, it is warranted to ask Concept Foundation to continue supporting promising manufacturers. The Concept Foundation strategy to identify suitable RH manufacturers and products is technically sound, but a more focused shift towards the most needed products and using a risk-based approach might be needed. In the QuRHM business case the technical barriers for generic manufacturers to achieve prequalification might (in the view of the RT) be slightly overrated, whereas the economic issues (lack of a real business case) and the downstream (country based) barriers are underplayed, and could benefit from more attention by Concept Foundation, RHSC and DFID. Value for Money in Concept Foundation and UNFPA is currently hard to assess. The RT did not get indications of DFID money being wasted. Concept Foundation reports, presentations and website are transparently open. Project management and internal communications can be strengthened. Concept Foundation needs to pay more managerial attention to its outsourced output areas 2-4. Collaboration with RHSC members could be intensified in both directions, especially towards generating better market and procurement information, and about activities in/with countries. Recommendations:  Concept Foundation to strengthen its management team to take care of the bigger workload, the outsourced work areas, and to clarify roles.  Concept Foundation to focus its support to manufacturers more on the most needed products, the most promising manufacturers and the procurers and governments who have the greatest capacity to shift the market.  Concept Foundation to monitor and report on the baseline and intermediate outcome indicators (price trends, volume, national registration, and market developments of the most needed RH medicines) in its annual reports (based on RH Interchange and its own intelligence).  Concept Foundation to review the standard MoU with manufacturers with a view of balancing confidentiality requirements by manufacturers with transparency.  Disaggregate the price indicator for COC because of the highly different price levels of the different products, which makes monitoring and reporting on average COC figures difficult.  UNFPA to implement a monitoring process for the implementation of the different steps in the ERP process  DFID to include VfM measures in its contract with PATH/Concept Foundation Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 39 6 WHO PQP AND CONCEPT FOUNDATION WORKING TOGETHER The disappointing results in terms of prequalified RH medicines during Phase II of the BMGF funded Quality Medicines for Reproductive Health programme (QMRH, 2006-2010) pointed to contextual factors that so far had not been adequately addressed: lack of incentives for RH manufacturers to apply for prequalification (the ‘business case’), and inadequate technical capacity of manufacturers to achieve adherence to WHO GMP standards and product dossier requirements within a reasonable time. At the same time the RHSC had funded the AQAS project over 15 months (12/2009 – 2/2011) implemented by Concept Foundation under oversight of the RHSC Market Development Approaches (MDA) Working Group (WG). This project aimed to “identify a group of hormonal contraceptive manufacturers with the capacity and intention to apply for prequalification of products by the World Health Organization (WHO) within a period of two years, and who are willing to provide low-cost, hormonal contraceptives of assured quality to meet less developed country commodity security requirements”47. Under this project a list of potentially interested and capable generic manufacturers was established; a GMP assessment and follow-up technical support was provided to selected companies; and collaboration with WHO PQP on promotional material and development of a business case was initiated. Towards the end of “Accessing Quality Assured RH Supplies” (AQAS) project, work on developing the follow-on QuRHM programme began. Both funders (BMGF and DFID) saw value in establishing more closer and formal linkages between the WHO PQP/RH and QuRHM programmes from the start, not only to increase efficiency but also to diminish risks: one programme is supposed to address the assumptions of the other, and success of both will depend on success of each of the individual programmes. Therefore BMGF and DFID also decided to have combined annual reviews of the two programmes. The diagram in Figure 5 below shows how the programmes intended to contribute to achieve their common outcome (focussing on hormonal contraceptives).48 47 Chinery L. AQAS Final Report. Concept Foundation: April 2011 48 A revised Theory of Change diagram including activities of both programmes had been included in the inception report and is attached in Annex 4 Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 40 Figure 5: WHO/PQP and QuRHM Programmes working together 6.1 Structures for coordination The QuRHM programme business plan specifies two management and oversight mechanisms which were set up shortly after commencement of the two programmes:  For operational monitoring and review the Programme Oversight Committee (POC) has been established with representatives of the two funders and the two implementing partners. The POC is also charged with overseeing coordination and ensuring complementarity is being achieved. The POC mechanism is also included in the BMGF supplement grant document.  For strategic oversight and direction the Technical Advisory Committee (TAC) has been established. The TAC draws its members from the relevant Reproductive Health Supplies Coalition (RHSC) working groups and other relevant RHSC members including WHO/QSM and the two funders. In addition, quarterly formal meetings were foreseen between the main implementing partners (Concept Foundation, WHO PQP, and UNFPA PSB) – 3 meetings were held between September 2011 and September 2012. Technical cooperation is governed by a MoU between WHO PQP and Concept Foundation that includes definition of the roles and responsibilities of the two partners (see Chapter 4 for more detail). # of aplications for PQ ↑; # of low-cost PQ Hormonal Contraceptives (HC) ↑ prices ↓ - procurement volumes ↑ access to & use of QA HC ↑ unmet need ↓ WHO PQP/RH QuRHM Programme business cases for & targeted TA to specific manufacturers; advocacy to donors & proc. agencies (RHCS partners) for harmonised QA policies; setting up and implementation of ERP process; advocacy & TA to public sector in pilot countries for implementation of QA standards; launch EOIs; develop specifications; dossier assessments; inspections; pre-audits; capacity building of manufacturers and NMRAs (Workshops etc.); organise ERP reviews; promote fast-track registration; business cases; transparent guidelines CPR ↑ MMR ↓ product failure rate ↓; cost-effectiveness ↑ Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 41 While the QuRHM logical framework, activity log and budget do not include specific activities related to coordination between the two programmes, WHO PQP/RH has as Objective 2 “Coordinate activities with the RHSC/Concept Foundation on WHO prequalification of RH medicines”. Budgeted activities are: regular meetings between WHO PQP and Concept Foundation, quarterly POC meetings, and a planning meeting to develop a joint monitoring & evaluation plan and table of responsibilities. Critical milestones are: (i) Planning & monitoring meeting between funders and implementers by November 2011 (Status: achieved) (ii) Joint monitoring & evaluation plan developed by WHO PQP and Concept Foundation by November 2011 (Status: not achieved; deadline extended and draft approved by funders in September 2012). 6.2 Assessment of achievement of joint outcomes for 2012 As per the TOR for this review WHO PQP/RH and QuRHM programme aim to prequalify 5-8 generic products (or achieve ERP approval) by end 2012, and the outcomes expected to be completed by WHO PQP and/or Concept Foundation by end 2012 were: 1. 7 additional API dossiers submitted to WHO PQP for review 2. The quality components of dossiers for 6-8 finished products submitted to WHO PQP for review 3. Two training workshops with generic manufacturers jointly conducted – one in China and one in India 4. 3 APIs and 2 finished products prequalified 5. Completion of baseline to provide information on how generic prequalified products impact on price reductions and volumes procured (based on UNFPA and other procurer tenders for OCs) 6. ERP and interim mechanism established and functioning 7. Workshop conducted to align procurement agencies on quality assurance policies (with UNFPA) We did not assess how the joint outcomes and targets were developed, but during our initial meeting with BMGF, WHO PQP and Concept Foundation in July 2012 we invited comments on the TOR and the joint outcomes were not questioned. There are some inconsistencies between the individual programmes’ targets for 2012 and those identified for the combined work: WHO PQP/RH specifies 5-8 prequalified products not including ERP recommended ones, 10 FPP applications submitted, and does not mention APIs specifically; QuRHM programme year 1 milestones specify 3 prequalified products plus 6 ERP recommended, 2 APIs prequalified, and 2 procurers’ workshops held. While these inconsistencies do at this point not considerably impact the achievement ratings for the combined programmes, any future definition of joint outcomes should be better aligned with the individual targets. Progress with achievement of the joint outcomes is documented in Table 10. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 42 Table 10: Status of joint outcomes for WHO/PQP RH and QuRHM programmes Outcome description Latest known status Comment Likelihood of achieving output by Dec 2012 Overall outcome 5-8 prequalified or ERP approved products 2 generic products prequalified; 12 ERP recommended Of the 12 ERP recommended by October 2012, 7 are generics from non-SRA manufacturers Already achieved Specific targets 1) 7 additional API dossiers submitted for PQ 5 API dossiers submitted CF expected 3 API dossiers being submitted during Q4 2012 Medium 2) Dossiers (QC) for 6-8 FPPs submitted 4 FPP dossiers received 3 passed screening; 1 under screening Medium 3) 2 jointly conducted training workshops 1 in China India workshop being planned for November 2012 High 4) 3 APIs and 2 FPPs prequalified 0 API; 2 FPPs 5 APIs under assessment for PQ with many major issues raised Already achieved for FPPs Low for APIs 5) Completion of baseline on potential price reductions Not available Updated baseline values not available; too early to assess? Low 6) ERP functioning 2 nd ERP concluded Already achieved 7) Procurers’ workshop on QA policies 1 conducted QuRHM programme specifies a target of 2 Already achieved The specific targets 1-4 and 6 are clearly dependent on how the two programmes performed individually and in their collaborative work. Achieving these targets depends on the effectiveness of advocacy and technical support work of Concept Foundation, the assessment and inspection efficiency and ways of engagement with manufacturers of WHO PQP, and how effectively the two programmes together planned, coordinated, communicated and monitored their interdependent work streams. Targets 5 and 7 seem to be more specific to the QuRHM programme. With regard to target 2 this might be achieved (for 6 submissions) by end 2012, because it seems that not all manufacturers that applied to the second ERP without already being in the WHO PQP pipeline have as yet complied with their obligation to submit their dossiers for prequalification49. Overall progress on the joint outcomes and targets appears better than 12-months progress of the individual programmes against their specific milestone targets. This is due to the formulation of the common targets and not necessarily because together the programmes perform better than individually. 6.3 Collaboration between the two programmes in practice WHO PQP and Concept Foundation together successfully prepared and implemented the manufacturers’ workshop in China, and a similar one in India is planned to be held shortly. These workshops also provide an opportunity for technical assistants contracted by Concept Foundation and technical experts of WHO PQP to communicate face to face. Concept Foundation technical experts also participated in a meeting organised by WHO for technical assistance providers to ensure 49 We draw this conclusion from information on # and type of dossiers currently under assessment – the names of manufacturers that are under assessment by WHO PQP are not made public. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 43 they are familiar with WHO standards and common problems experienced in terms of dossier and GMP deficiencies. Based on the MoU, Concept Foundation and WHO PQP have frequent contacts to address technical and capacity building issues. Being based in Geneva the Concept Foundation CEO regularly visits WHO PQP to discuss technical issues arising from their work with manufacturers with the WHO PQP head assessor and inspector, and content and planning issues related to common capacity building activities with the WHO PQP head of capacity building & training. The fact that these contacts are being described as ‘informal’ by both parties imply two things: (i) interpersonal relationships are good – ad hoc meetings are possible and take place without administrative constraints; and (ii) a more structured approach might be missing that could assist in getting most out of these interactions and better realise synergies. In fact, development of the joint monitoring and evaluation plan (critical milestone for Objective 2 of the WHO PQP/RH) that could be part of such a structured approach was delayed several times. While it was agreed as a priority activity during POC meetings implementing partners found it difficult to plan / allocate time for related meetings due to time constraints and other priorities. A draft plan developed by Concept Foundation was made available to us (version 9 September), which foresees joint monitoring at two levels:  Level 1 links WHO PQP/RH Objective 1 critical milestones with QuRHM Programme Output 1 and 2 indicators and milestones. Progress is suggested to be monitored monthly by the two programmes and quarterly by the POC.  Level 2 shows linked or mutual activities at activity milestone level. Progress is suggested to be monitored as for Level 1 plus through bi-weekly meetings of WHO PQP and Concept Foundation staff. Red flags are indicated for two activity areas: joint development of business case (QuRHM: no planned activities) and joint organisation of 2 manufacturers workshops (QuRHM: no specific budget allocation) The draft framework does not include WHO PQP assessment and GMP activities under linked activities. It therefore does not yet provide for linkages between the key processes of the two programmes that would lead to product prequalification, for example how specific manufacturers’ initial assessments, technical support, improvement action plans (Concept Foundation) are linked to submission of dossiers and subsequent dossier assessment and inspection processes, and ultimately prequalification (WHO PQP). This could be quite time consuming but has the potential to provide valuable information on how/whether collaboration between the programmes in these areas need to be improved to accelerate the overall process of prequalification. Joint development of the business case for manufacturers to engage in prequalification or SRA approval was reported on in the initial POC meeting in November 2011 and the first WHO PQP/RH progress report (March 2012). Due to lack of progress a specific meeting was held in July 2012 where it became clear that there are differences in perceptions of WHO PQP and Concept Foundation regarding the critical areas of focus (see Chapter4). Having the two programmes working separately on this priority area (as is happening now) still requires collaboration in order to ensure complementarity and avoid waste of resources. 6.4 Joint governance of WHO PQP/RH and QuRHM Programme The Programme Oversight Committee (POC) is the immediate governance mechanism for the two programmes that has been foreseen in both programme documents. TORs establishing POC roles Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 44 and responsibilities, membership, chair, and meeting schedules are available. The POC is supposed to meet at least quarterly, two meetings to be held face to face and the others using teleconferences. With regard to POC membership the October 2012 POC meeting agreed to add UNFPA PSB as a member being the principal for the ERP Secretariat and the procurement meetings/discussions. The RT noted that RHSC (which was envisaged in the DFID Business Case to have oversight responsibility) is not (yet) a member of the POC. During the review period 3 meetings were held (November 2011, and February and July 2012). A fourth meeting was held in October 2012 during the RHSC annual membership meeting. The POC meeting minutes available to the RT (first and third meeting) provide evidence that relevant topics are being discussed and action points identified. However, it is not clear from the minutes whether all agreed action points are followed up in subsequent meetings. For example, in the first POC meeting Concept Foundation was requested to clarify its prioritization criteria for products and companies to work with, but minutes of subsequent meetings do not address this point. Most direct stakeholders interviewed consider the POC meetings useful for providing a formal opportunity to discuss joint programme issues, and get clarification on the two funders’ expectations. However, there are also perceptions that specific coordination mechanisms are imposed by the funders. The Technical Assistance Committee (TAC) is a more strategic governance mechanism initially set up under the QuRHM programme but with representation of WHO/QSM. The working of the TAC is described in more detail in Chapter 5. In general both governance mechanisms have their place but are not used to their full potential. The TAC could be used better to benefit from the broad experience of the diverse RHSC membership through strategic selection of members, and working according to clear TOR. For the POC the joint monitoring and evaluation plan could provide a good opportunity to focus meetings and foster understanding of the interdependency of the programmes. Risks to both programmes are apparently not being explicitly discussed during POC meetings (although included in the POC TOR). At the TAC meeting in October 2012 external factors impacting on the programmes’ goals were touched on, and some useful suggestions for addressing these were made. The expertise at TAC level could be better used to identify external risks, realistic options for risk mitigating strategies, and related existing activities by RHSC members. At POC level identified risks and effectiveness of mitigating measures should be monitored on a regular basis (as a fixed agenda item) so that necessary changes in the programmes’ directions can be agreed timely. 6.5 Challenges and lessons learnt Several challenges in terms of effective coordination and collaboration were identified by interviewees involved in the implementation of the two programmes, and by the RT:  The different way the programmes have been conceived (a logical framework with four output areas versus a project framework with two objectives); this makes development of a joint monitoring tool difficult  Different timeframes (WHO PQP/RH 15 months, ending Dec 2012; QuRHM 3 years, ending March 2014); the perspective on planning joint activities beyond end 2012 is missing Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 45  Different institutional environments: Concept Foundation a relatively small and flexible NGO, and WHO PQP situated in a big UN organisation mandated by member states and with the related administrative constraints  Time required to be allocated to structured collaboration in the context of existing capacity constraints (time wise)  Insufficient capacity of implementing partners to establish consensus in areas were differences of opinions surface (for example, the business case or joint monitoring system)  Not all joint activities are included in the individual programme plans or logframe It is not easy to have two organisations with already established programmes (WHO PQP, and Concept foundation with several years of previous engagement in promoting quality assurance for RH medicines and working with manufacturers) starting to working together in more than a consultative manner. Structured collaboration requires commitments and needs to show clear benefits to all parties. 6.6 Conclusions and recommendations While progress with achieving the common outputs for 2012 is positive, there is no clear evidence yet how collaboration between WHO PQP and Concept Foundation contributed to this, and what the contribution of the individual programmes is. Assuming that WHO PQP/RH will receive supplementary funding by BMGF for 2013 the following is recommended: 1. Both programmes to improve the structure, reporting and follow-up of monthly collaboration meetings 2. Activities to be implemented together to feature in the implementation plans of both programmes with indication of what each programme contributes in terms of funding 3. Development of a joint work plan for activities that are to be implemented together (agree on joint activities, targets, and who does what separately) 4. Agree on joint outputs and targets to be achieved by end 2013 (also in view of the 2013 Annual Review, and the 2014/15 impact evaluation) 5. Improve efficiency of the POC (e.g., ensure structured follow up on decisions made at previous meetings) 6. Address internal and external risks at POC and TAC meetings 7. Agree on purpose and frequency of regular formal meetings between WHO PQP and Concept Foundation staff focussing on the benefit these provide to implementing partners and realistic assessment of feasibility in terms of time commitments 8. Consider setting up a monitoring system that would allow how WHO PQP and Concept Foundation activities interact and impact progress of a product through the prequalification chain (from submission to prequalification) 7 RISKS, EXTERNAL FACTORS AND OVERALL CHALLENGES The RT assessed whether the assumptions and (external) risks in the original project documents had changed, or whether new risks had arisen. The following documents were consulted to identify all known risks:  Assumptions in the QuRHM logframe Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 46  Risk section in the QuRHM Business Case and Annual Review template  Assumptions and external factors listed in the WHO PQP funding proposal to BMGF  Additional risks and external factors that the RT has encountered in interviews The QuRHM logframe states the following assumptions: Table 11: Assessment of QuRHM programmes assumptions Level/ area Assumption Risk rating Comment purpose This programme focused on supply side but not demand side interventions is unlikely to deliver changes to CPR and unintended pregnancy alone. However significant work is done by DFID and other donors to improve access to and quality of family planning services – of which this work is a part. N/A RT agrees with the diagnosis of the biased focus. Compensating this focus will require more collaboration integration of efforts between CF and RHSC members, and between the QuRHM and other RH projects operating in countries. output 1 Co-operation of/investments by manufacturers of FPPs and APIs High Depends on the (economic) incentives which are indeed beyond the project. But CF can make improved market info available in the Business Case being developed. Capacity of WHO to process dossiers remains This depends on WHO PQP being funded by the funder. Can be controlled by funder output 2 Advocacy efforts with senior management of procurers are successful Medium, changed to Low Procurers are likely to agree to a QA policy. Problem is they need to have sufficient QA products to procure that are also registered in their target countries. Low cost prequalified RH medicines available to purchase Medium, changed to High New problem: 2 prequalified COCs are unfortunately more costly than existing products; this should not be unexpected as the new entrants’ volumes may be smaller than traditional suppliers. Pushing costs too low might de-incentivize manufacturers to go for high quality standards. UNITAID experience shows that you sometimes have to pay more in the short term to new entrants in order to achieve longer term supply diversity and reduced prices (LPV/r is a classic example). Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 47 Level/ area Assumption Risk rating Comment output 3 High level national stakeholders are receptive to revised approaches for quality assurance Medium Underestimated problem in project design: will need more attention and collaboration with national RHSC members to be resolved output 4 Advocacy efforts are able to ensure continuing visibility and stakeholder interest around QA issues Low RHSC members have sufficient interest in quality issue. CF to increase its activities. Risks in annex E of the QuRHM business case can be modified as follows: Table 12: Recommendations on QuRHM risk rating modification Risk Rating Comments Lack of supplier willingness to accept TA probability from Medium to Low RT could not confirm this fear with manufacturers. Wrong risk definition? Risks lie in commitment of manufacturer to proceed through prequalification and finally offer their products in the target countries markets perceived lack of transparency of WHO process probability from High to Medium RT could not confirm this perception; according RT WHO is very transparent but more efforts needed to communicate this effectively procurement agencies do not agree/abide by quality policy probability Medium Mixed answer: procurers are very willing to accept QA policy (low risk), but they need to have 2 or more QA products to buy (high risk) as prequalification does not equal availability to local markets more PQ products does not lower prices probability increased from Low to Medium Recently Prequalified COCs are more costly than some originator products but sample size very small. The assumptions in the WHO PQP project narrative50 are less serious risks, and not always “external” risks; some of them are actually tasks for WHO, and several are within control of WHO PQP or Concept Foundation: 50 19 August 2011 supplementary funding request by WHO PQP to BMGF. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 48 Table 13: Assessment of WHO PQP/RH assumptions Assumption Comment That commitment of e.g. Reproductive Health Supplies Coalition to ensuring increased access to quality- assured RH medicines is maintained Very low risk – this is the primary goal of RHSC. That a vigorous market for quality-assured hormonal contraceptives and other RH medicines can be stimulated and maintained over time Medium risk – public sector funds are increasing - see July 2012 FP Summit commitments by donors, but impact on private sector less sure. Information about the market can be collated by QuRHM in close collaboration with RHSC members, and shared with manufacturers. That procurers adhere to quality assurance policies and procure only quality-assured RH medicines Mixed low and high risks. Output 2 in the QuRHM project addresses this; procurers are probably willing to adhere to QA policy, but can only do so if there are 2 or more PQ or ERP recommended products per category, and even then this does not mean that these products will be made available by manufacturers to all local market of interest. That a business case for production of contraceptives is sufficient to convince generic contraceptive manufacturers to invest in producing quality-assured products High risk - Business case content being developed by WHO and CF may mitigate risk, but convincing RH manufacturers might not be easy. Much will depend on building capacity of NMRAs in low-income countries to ensure standards of marketed products, and on wider political support in these countries for quality assured medicines vs. local industrial policies. That communications with manufacturers are clear and consistent and ensure their understanding of requirements for prequalification Is part of WHO PQP’s tasks (not an external risk) That RH medicines manufacturers will be able to meet the requirements of the WHO PQP to obtain prequalification Is task for Concept Foundation in QuRHM; can also be managed by WHO PQP. That PQP will receive sufficient numbers of product dossiers for evaluation Medium risk, but relates back to business case. In addition, both WHO PQP and Concept Foundation can try to actively ‘sell’ PQP to selected manufacturers. Project loses its rationale if no dossiers are received. That the product dossiers submitted will be of sufficient quality to enable them to become WHO-prequalified within a reasonable time frame Is task for CF in QuRHM, and task of WHO PQP in providing training and feedback That WHO will prequalify sufficient products in each category on the PQP expression of interest list to stimulate price competition and lower prices Is related to the previous risk. Is a risk for RHSC and the project outcome Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 49 Assumption Comment Project funding for both PQP and Concept Foundation is available at the same time, so that project coordination can be assured from the start of activities by both entities Very likely; is the intention of both DFID and BMGF Staff time and schedules will permit close collaboration Can be managed by CF and WHO PQP; not an external risk In addition to the identified risks in project documents, the RT would like to discuss some additional risks. 7.1 PQ manufacturer not interested in specific markets, sectors or tenders The manufacturer of a successful prequalified product might not be interested in all markets. For example he might only be interested in middle income countries with sizable private sectors, or in rich countries, but not in tenders (large international buyers or public sector tenders by national medical stores). The purpose of PQ can be addressed in the business case(s). Low interest in public sector markets in low-income countries should be spotted early in Concept Foundation’s support to manufacturers, so that they can focus on manufacturers that have potential and willingness to achieve QuRHM goals. 7.2 Supply side focus Both WHO/PQP and QuRHM project are focusing mainly on (generic) manufacturers and the quality of their products (supply side), while they have limited roles at country level (demand side). Achieving a list of prequalified RH medicines alone is unlikely to achieve changes in CPR or unintended pregnancies. Demand side market shaping actions also influence incentives for manufactures to get engaged in prequalification in the first place. Figure 6 presents some examples for this. Overall the focus on push interventions at the supply side poses risks for both projects, which need to be mitigated or managed. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 50 Figure 6: Examples for supply and demand side criteria providing incentives for prequalification 7.3 Downstream risks Most of the risks of achieving impact will be downstream at country level: the current projects are not very focused on this level, and it is also beyond the scope and not within the comparative advantage of both WHO PQP/RH and QuRHM programmes to comprehensively work at downstream level. Even if sufficient prequalified or ERP recommended products become available, it will take considerable time before these quality assured products can and are actually used by women. This will affect the impact evaluation currently planned for 2014/5. WHO and Concept Foundation will need to mobilize the help of national RHSC members and closely liaise with existing downstream activities at country level (e.g. in the context of national Sexual and Reproductive Health Rights Strategies, Contraceptives Commodity Security Strategies, or overall health commodity supply chain strengthening initiatives) to try and mitigate all the below risks: National registration Prequalified products must – like all other medicines – be registered in each country where they are needed. Although most NMRAs will accept prequalification as a strong recommendation, formal procedures must be adhered to (local applicant, submission of documents (not always CTD format), payment of application fees, package insert adapted to local requirements, etc.). Some countries will allow a “light” evaluation because of the PQ status, and others will allow a “fast-track” process for essential medicines, but this is no guarantee that the process is fast. National registration procedures can take anything from 3 Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 51 months to 5 years51. Both WHO PQP and Concept Foundation have activities planned to address this, so it is not an external risk. NMRAs have agreed to work with WHO on fast- tracking essential medicines. Concept Foundation could also work with a local RHSC member with professional contacts in national medicines regulation. National Procurement – a story of what could happen Once registered, and assuming that it is already on the National Essential Medicines List, the prequalified product would need to enter the procurement cycle – which can take a long time depending on the frequency of tendering. There will most likely be competition of other registered products that are not (yet) prequalified, so companies will also need to do local market research before they make an offer in a tender. Once selected as a winning bid, it might take some time to allow for the official complaints procedure before a contract is issued. Some countries then adjust the quantities needed, so manufacturing, packaging and delivery will only start after firm orders have been received. Import procedures might also delay delivery. Procurement staff and programmes (and even consumers) might have historic preferences for specific products, so just being the cheapest and quality assured is no guarantee for being bought. After arrival in the National Medical Stores or at a national wholesaler, the product might be quarantined for some time to permit quality laboratory testing. After confirmation, the product will go on official sale. The pipeline (regional, district stores) will need to be filled (first expired, first out) before it finally arrives in a health facility where it can be dispensed to the consumer only after the old stock is finished. Seeing impact at the patient level can take much more time than envisaged. To allow all this to happen, the RT recommends to delay the impact evaluation as long as is permitted but at least until 2015. 8 OVERALL CONCLUSIONS AND RECOMMENDATIONS 8.1 Conclusions It is still too early to confirm or reject the hypothesis behind both projects: That an increased number of quality-assured generic products can create more competition between quality-assured product manufacturers, lower prices, offer more choice to procurers and women, and satisfy unmet need. The disappointing number of only two prequalified generic hormonal contraceptives is mitigated by the 12 generic products recently having been recommended in the ERP process. The continued absence of quality assured generic implants or injectable hormonal contraceptives remains a worry, as reliance on originator companies is being maintained. Prices of two of the three newly prequalified products are not (yet) lower than originator products, and one of them is not yet widely registered in target countries. The prequalified or ERP recommended products have not (yet) led to measurable impact in target countries. 51 Assessment of medicines regulatory systems in sub-Saharan African countries. An overview of findings from 26 assessment reports. WHO, 2010. http://www.who.int/healthsystems/Assessment26African_countries.pdf Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 52 The WHO/PQP RH and QuRHM programme designs are primarily focused on the supply (push) and regulatory aspects, rather than the demand (pull) side. A comprehensive assessment of the total RH product market and its characteristics was apparently not done before embarking on prequalification of RH medicines. Expectations for the programmes were overambitious probably based on successes of PQP of HIV/AIDS medicines (prequalification of TB and Malaria medicines was less successful, especially in the beginning), and fuelled by the pressure to show successes at the global Family Planning Summit held July 2012 in London. It is now understood that generic RH product manufacturers in low- and middle-income countries need more time to implement quality improvements and that incentives for doing so are not as obvious as anticipated. WHO PQP has worked hard to make its PQP for RH a well-run operation. It is doing a good technical job and adheres to SOPs and KPIs, while being open for criticism and trying to address transparency and communication issues where possible. WHO PQP has regular meetings with manufacturers, but could improve its communication with other stakeholders, and proactively promote its programme. “PQ needs to be demystified.” WHO also tries to build capacity in national Medicines Regulatory Authorities in low-income countries, and takes their inspectors along on GMP inspections and dossier assessments. Despite the hard work, the number of prequalified generic RH medicines has remained disappointingly low: the number of applications for prequalification is much less than expected, and those received still need a long time to proceed to prequalification due to the required improvements in quality of dossiers and GMP. Concept Foundation is making steady progress in its TA work with generic companies, although details on these engagements were not accessed by the RT. Concept Foundation is optimistic about the speed by which it can support generic companies and deliver them to the WHO PQP for assessment. The four generic companies, who volunteered information to the RT on Concept Foundation’s work, are generally satisfied with the support they get. Concept Foundation has been paying less attention to the downstream activities, which it has contracted out to UNFPA and consultants. RHSC (members) should probably take more responsibility here, especially in working in/with countries. Over the review period, Concept Foundation’s capacity to deliver or supervise all outputs was stretched, also because of additional and unplanned demands put on the organisation in the context of the July 2012 Family Planning Summit and related engagement with the UN Commission for Life-Saving Medicines. Internal management is now being strengthened, so that Concept Foundation can properly supervise its outsourced activities. Donors DFID and BMGF made a wise decision to have the two programmes work together. WHO PQP and Concept Foundation’s work is complementary, and both are needed to get more quality assured RH medicines to the market. The coordination structures could work more effectively, and most of the good collaboration was achieved by ad-hoc visits and personal relations. A joint monitoring plan has been developed but has not yet had the time to show results. A joint Programme Oversight Committee (POC) meets regularly and enthusiastically, but makes largely ad- hoc decisions and does not follow-up on decisions of previous meetings or provide structural supervision and risk monitoring. The latter would be a priority taking into consideration the many external factors that impact on the programmes’ successes. WHO PQP and Concept Foundation have good technical skills for the regulatory and TA work, but less so for developing business cases or addressing downstream issues. Their output/milestone rating for year 1 is lower than expected due to delays in contract signing, and an overly ambitious timetable in the project design. Results of the forthcoming business case work should provide an improved evidence base upon which to better focus and tailor the grantees’ efforts towards those that have the most potential for market shaping. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 53 Assessing value for money (VfM) has been a challenge. Concept Foundation was not asked by DFID in its contract to report on VfM. The subcontract with UNFPA could be clearer in describing outputs; efficiency and VfM were difficult to assess, also due to the UN rules of the “Single Audit System”. The ERP process was helpful in generating 12 more quality assured sources for procurers. Concept Foundations’ stakeholder management can be strengthened, and overall responsiveness improved. It could also make better use of the Technical Advisory Committee, which was supposed to ensure links with the wider RHSC community. RHSC (through TAC) has played only a limited advisory or supportive role. RHSC members are not yet much involved in downstream activities at country level. After this first Annual Review it is still too early to reject or confirm the principle hypothesis. Looking at the future, the RT notes that the RH medicines pipeline contains much needed products. The RT is confident that both programmes have the technical ability to deliver the outputs when given more time and with an enhanced evidence base to help tailor efforts. For the generic manufacturers, a more convincing business case will have to be presented. RHSC should prioritize its work in/with countries. Women in low- and middle-income countries can still be hopeful to see more quality assured RH medicines in the future. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 54 8.2 Overall recommendations (Specific recommendations were made in specific chapters) To BMGF:  To continue supporting WHO PQP after the end of the current project phase  To review the existing project framework table and develop any new programme using SMART indicators and milestones that are clearly linked to the project goal and objectives  To consider joint funding mechanisms and/or harmonised reporting procedures with other donors (e.g. UNITAID, Global Fund) to increase efficiency and make best use of scarce resources at WHO PQP To DFID:  To assess how overall oversight of the QuRHM programme by the RHSC can be improved  To consider whether Value for Money reporting requirements can be added to the current QuRHM contract To the joint donors:  Using institutionalised RHSC mechanisms, to improve the involvement and support of RHSC with a specific focus on downstream activities and demand side in countries  To ensure that the POC is becoming more structured and effective  To consider funding of surveys of actual RH medicines quality for building the evidence base required for further promoting the need for quality assured RH medicines To WHO PQP:  To improve its overall visibility and make its key performance indicators public To the Concept Foundation:  To reorganize and expand its management team to enable it to respond more adequately to the increased workload and demands, and to clarify the lines of command among top management.  To review the terms of their standard MoU with manufacturers in view of increasing transparency while providing the minimum level of confidentiality required to have manufacturers engaging with the programme To the contracted partners jointly:  To jointly plan, monitor and evaluate their joint activities  To institute a mechanism that allows linking outcomes of Concept Foundation technical support activities to manufacturers’ progress in the WHO prequalification process  To consider how downstream issues affect success of their programmes, and to make proposals for relevant linkages with existing programmes in countries  To use the results of the business case development to better prioritise their investments To the RHSC:  RHSC, RHSC Secretariat and MDA working group to actively engage with Concept Foundation to support the work in/with countries  To clarify with Concept Foundation the roles, responsibilities, and membership criteria of TAC, and to improve mutual communication channels Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 55 ANNEXES Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 56 ANNEX 1. TERMS OF REFERENCE – VERSION 3 AUGUST 2012 Independent Monitoring, Review & Evaluation of WHO Prequalification Programme and Quality for Reproductive Health Medicines Programme52 Overview  Document Purpose The purpose of these Terms of Reference (“TOR”) is to for the Bill & Melinda Gates Foundation (the “Foundation”) and the UK Department for International Development (“DFID”) to inform a consultant (“Consultant”) for the design and implementation of an independent, external review of two programs funded by the Foundation and DFID, the WHO Prequalification Programme (PQP) for Reproductive Health (RH) Medicines and the Quality for Reproductive Health Medicines (QuRHM) program. The Consultant will provide two comprehensive organizational and program reviews (2012 and 2013), and a final evaluation of the QuRHM program to take place at the end of the program (2014/15).  About the Bill & Melinda Gates Foundation Guided by the belief that every life has equal value, the Bill & Melinda Gates Foundation works to help all people lead healthy, productive lives. In low- and middle-income countries, it focuses on improving people’s health and giving them the chance to lift themselves out of hunger and extreme poverty. In the United States, it seeks to ensure that all people—especially those with the fewest resources—have access to the opportunities they need to succeed in school and life. Based in Seattle, the foundation is led by CEO Jeff Raikes and co-chair William H. Gates Sr., under the direction of Bill and Melinda Gates and Warren Buffett.  Project Background Lack of quality, affordable contraceptives is a major barrier to women accessing modern methods of contraception. Currently the market for quality hormonal contraceptives (oral pills, injectables and implants) is dominated by originator companies whose products are costly. Cheaper products are available from generic companies; however they are not always of sufficiently high quality to ensure efficacy and safety. The same issue affects the quality and cost of other essential RH medicines such as oxytocin and misoprostol. Yet through the World Health Organization’s (WHO) PQP, working with UNFPA, condoms and IUDs have successfully been prequalified. With more competition, prices have fallen while quality has been maintained. In 2006 the WHO PQP called for expressions of interest for prequalification of contraceptives to encourage generic hormonal suppliers to gain approval for quality products and hence access to the international public sector market (e.g. procurement by UNFPA), thereby improving supplier choice and competition for key products. However, to date, only two generic hormonal contraceptives or other RH medicines have been approved and no additional manufacturers for these medicines have submitted dossiers to the WHO PQP. Barriers to suppliers embarking upon or completing the process include:  Limited understanding of the business benefits of prequalification;  Understanding and meeting the challenging technical needs to bring products to a high enough standard for product approval; 52 This TOR includes amendments made at the 24 July 2012 kick-off meeting in Geneva and additional questions by BMGF. Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 57  Some major funders and procurers do not yet insist on product approval by WHO or a Stringent Drug Regulatory Authority53 (SRA); and  Perception that the WHO prequalification process is not transparent and is overly stringent54. The Foundation has provided nearly $3.5 million USD over the last three years to the WHO prequalification scheme for reproductive health medicines. The Foundation has funded an additional $1.5 million USD for 15 months to support the prequalification of additional RH medicines with a focus on hormonal contraceptives (with the possibility of extending beyond this period). The aim of this work is to:  Ensure transparency of guidelines and requirements for prequalification and the Expert Review Panel (ERP);  Engage with applicants to ensure understanding and agreement on corrective actions that need to be undertaken;  Help build the business case for manufacturers to obtain WHO prequalification; and  Review dossiers, conduct inspections, and respond to applicants in a reasonable time. DFID has been requested through the Reproductive Health Supplies Coalition (RHSC) to fund a complementary work program, the Quality for Reproductive Health Medicines (QuRHM) by the Concept Foundation for £3.2M over three years to help systematically tackle the barriers above by:  Working with generic manufacturers to encourage and enable them to prequalify products through the WHO PQP;  Engaging with public sector and other major procurers to develop and promote adherence to the UNFPA quality assurance policy55;  Encouraging relevant authorities in low- and middle-income country governments to adopt and use internationally agreed quality criteria in their own procurement and tendering processes; and  Raising awareness among donors of quality issues in relation to contraceptives and other RH medicine. The program activities supported by the Foundation and DFID with WHO and the Concept Foundation are interdependent and mutually supportive of each other. DFID and the Foundation are working in close partnership to ensure that adequate communications and operational links are made – including regular joint meetings with WHO, Concept Foundation and both donors; sharing of project documentation and commissioning joint annual reviews and a final evaluation (through a joint committee, the Programme Oversight Committee, or POC). The work also benefits from advisory inputs from a wider Technical Advisory Committee (TAC) for members of the RHSC with interests in developing a quality sustainable market. Working together, these two programs aim to prequalify 5-8 generic products (or achieve interim, External Review Panel approval) by the end of 2012 with additional targets for subsequent years based on initial performance. Seven outcomes to be completed by WHO PQP and/or Concept Foundation by the end of 2012 are: 53 Stringent Drug Regulatory Authority means a regulatory authority which is: a member of the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH); or an ICH observer, being the European Free Trade Association (EFTA) as represented by Swiss Medic, Health Canada and WHO; or a regulatory authority associated with an ICH member through a legally binding mutual recognition agreement including Australia, Norway, Iceland and Liechtenstein. 54 As reported in McKinsey & Company review commissioned by FOUNDATION 2011. 55 www.unfpa.org/./quality%20assurance/UNFPA%20QA%20Policy%20for%20RH%20medicines_31March11.pdf Accessed 22nd June 2011 Independent Monitoring, Review & Evaluation of WHO/PQP/RH & QuRHM HERA / Final Annual Review Report 2012 / 30 November 2012 58 1. 7 additional API dossiers submitted to WHO PQP for review 2. The quality components of dossiers for 6-8 finished products submitted to WHO PQP for review 3. Two training workshops with generic manufacturers jointly conducted – one in China and one in India 4. 3 APIs and 2 finished products prequalified 5. Completion of baseline to provide information on how generic prequalified products impact on price reductions and volumes procured (based on UNFPA and other procurer tenders for OCs) 6. ERP and interim mechanism established and functioning 7. Workshop conducted to align procurement agencies on quality assurance policies (with UNFPA)  Project Scope The Consultant will design (or build on an initial evaluation of PQP) and implement a process to independently and externally review both the Foundation funded WHO PQP for RH medicines and the DFID funded QuRHM program. This will provide two comprehensive organizational and program reviews (2012 and 2013), and a final evaluation of the QuRHM program to take place at the end of the program (2014/15) for a total of 3 reviews over 3 years. The Foundation’s contract will specifically cover the first two program reviews (2012 and 2013) and DFID’s contract will cover the evaluation components in 2014 or 2015. The purpose of this external evaluation is to gain insight on progress toward the goal of prequalifying 5-8 generic RH products by the end of 2012 from a baseline of 0 at end 2010, and the longer-term goal of 25 generic products produced by 15 manufacturers, and 7 active pharmaceutical ingredients by the end of 2014, given internal and external factors. It is also anticipated that through the QuRHM program, a total of six major procurers (including social marketing organizations), which are members of RHSC, will have agreed and adopted common quality assurance definitions, and policies for purchasing essential RH medicines. Four developing country governments will have accepted and be using internationally agreed quality criteria in their own procurement and tendering processes. It is anticipated that the consultant will build on (and not duplicate) efforts undertaken in a recent evaluation of the PQP for HIV, Malaria and TB medicines and look at specific successes and failures for t

View the publication