Creating a Culture of Evidence Use Using an Innovative Knowledge Translation Platform to Inform HIV/AIDS Programming for Women and Girls

Publication date: 2017

Creating a Culture of Evidence Use Using an Innovative Knowledge Translation Platform to Inform HIV/AIDS Programming for Women and Girls W O R K IN G P A P E R DECEMBER 2017 Jill Gay Melanie Croce-Galis Karen Hardee Amelia Peltz Creating a Culture of Evidence Use Using an Innovative Knowledge Translation Platform to Inform HIV/AIDS Programming for Women and Girls DECEMBER 2017 Jill Gay, What Works Association Melanie Croce-Galis, What Works Association Karen Hardee, Population Council Amelia Peltz, USAID The Evidence Project Population Council 4301 Connecticut Avenue, NW, Suite 280 Washington, DC 20008 USA tel +1 202 237 9400 evidenceproject.popcouncil.org The Evidence Project is made possible by the generous support of the American people through the United States Agency for International Development (USAID) under the terms of cooperative agreement no. AID-OAA-A-13-00087. The contents of this document are the sole responsibility of the Evidence Project and Population Council and do not necessarily reflect the views of USAID or the United States Government. The Evidence Project uses implementation science—the strategic generation, translation, and use of evidence—to strengthen and scale up family planning and reproductive health programs to reduce unintended pregnancies worldwide. The Evidence Project is led by the Population Council in partnership with INDEPTH Network, International Planned Parenthood Federation, PATH, Population Reference Bureau, and a University Research Network. What Works Association aims to improve the lives of women and men through evidence-based guidance on policies and programs that promote gender equality. Published in 2017. Suggested citation: Gay, Jill, Melanie Croce-Galis, Karen Hardee, Amelia Peltz. 2017. “Creating a Culture of Evidence Use: Using an Innovative Knowledge Translation Platform to Inform HIV/AIDS Programming for Women and Girls,” Working Paper. Washington, DC: Population Council, The Evidence Project. Photo credit on cover page: © 2006 PATH/Mike Wang, Courtesy of Photoshare © 2017. The Population Council, Inc. Table of Contents ACKNOWLEDGMENTS . vi LIST OF ACRONYMS . vii ABSTRACT . viii INTRODUCTION . 1 METHODOLOGY .2 RESULTS . 7 DISCUSSION .20 LIMITATIONS .23 CONCLUSION .24 REFERENCES .25 vi CREATING A CULTURE OF EVIDENCE USE Acknowledgments This paper greatly benefited from interviews with key informants from UN agencies, donor organizations, civil society organizations and government agencies. The authors are grateful to Christina Memmott for her research support for this work and Anneka Van Scoyoc for production of the paper. THE EVIDENCE PROJECT vi i List of Acronyms American Public Health Association Concept Note Country Operational Plan Civil Society Organization The United Kingdom Department for International Development Gender Assessment Gender Assessment Tool Gender Based Violence International AIDS Conference London School of Hygiene and Tropical Medicine National Strategic Plan Lower and Middle Income Countries Office of the Global AIDS Coordinator Open Society Foundation Opioid Substitution Therapy Population Association of America Pre-exposure Prophylaxis Sexual and Reproductive Health Technical Review Panel Joint United Nations Programme on HIV/AIDS United Nations Development Programme U.S. Agency for International Development APHA CN COP CSO DFID GA GAT GBV IAC LSHTM NSP LMIC OGAC OSF OST PAA PEP SRH TRP UNAIDS UNDP USAID viii CREATING A CULTURE OF EVIDENCE USE Abstract Given that moving evidence from a study to policy, programs and practice can often take a decade or more, what is the best way to get evidence in the hands of those developing policies and programs to speed its use? Enhancing the use of evidence in policies and programs through an innovative web-based knowledge platform, What Works for Women: Evidence for HIV/AIDS Interventions (www.whatworksforwomen.org), resulted in major changes in National Strategic HIV Plans plus Concept Notes submitted to the Global Fund for AIDS, TB and Malaria in fourteen countries in Africa, Asia, and Latin America. Additionally, the What Works platform has been a key resource for key institutions in the HIV pandemic: PEPFAR, Global Fund, UNAIDS and civil society organizations. Lessons learned from the creation and dissemination of this knowledge platform may be useful for many other potential applications to increasing evidence-informed, gender responsive policies. THE EVIDENCE PROJECT 1 Introduction It is clear that public health policies and programs to address HIV and AIDS should be informed by evidence (Piot, 2015); what is less clear is how to ensure that policy makers and other stakeholders have access to the relevant scientific evidence as they are deliberating policies and developing programs. While it is unlikely that policy and program decisions will be based solely on evidence from research (Kim, 2006); (Eyben, 2013); (DFID, 2014), the absence of strong research evidence also “makes it unlikely that government will adopt an innovation” (Spicer et al., 2014). Yet moving evidence from a study to policy, programs and practice can often take a decade or more (Dickson et al., 2011). For policy makers and programmers in Lower and Middle Income Countries (LMIC), accessing evidence from studies in the peer-reviewed literature can be costly, complicated and time consuming (Lavis et al., 2009). While initiatives to increase open access to journals are useful, abstracts of articles, which are freely available, may not have enough information to be able to understand the key elements in the program that led to successful outcomes. Furthermore, while individual studies are important, to inform programming, syntheses of bodies of evidence tend to be more useful (DFID, 2014). What is the best way to get evidence in the hands of those developing policies and programs? Based on a review about how evidence is used in policy, program and practice decision-making, Hardee and Wright (Hardee and Wright, 2015) identify building cultures of evidence use as one of five categories of interventions to enhance the contribution of research to decision-making. Interventions to enhance cultures of evidence use have included strengthening capacity for evidence use; using intermediaries between researchers and decision-makers; building knowledge translation platforms; supporting rapid response mechanisms to provide evidence; making research directly available; and better packaging and communication of findings. This paper investigates the outcome of an initiative to enhance use of evidence on HIV and AIDS programming for women and girls, using the components of building cultures of evidence use identified by Hardee and Wright (Hardee and Wright, 2015). The initiative, What Works for Women and Girls: Evidence for HIV/AIDS Interventions (hereafter What Works) that became www.whatworksforwomen.org, is a comprehensive review of evidence from nearly 100 countries of interventions for prevention, treatment, care and support to strengthening the enabling environment for which there is evidence of success (see Box 1). Launched in 2010, the website serves as a type of knowledge translation platform, and aims to make research available to users in an easily accessed format focused on policy and program implications of the research. In 2011, the website won the Gold Award for electronic publications from the Council on Foundations’ Wilmer Shields Award for Excellence in Communications for making a wealth of information easily accessible in a reader-friendly format. The website was also highlighted in an article by Heidari et al. (2011) as a compilation of important information on gender and HIV that made access to knowledge more widely available. Since tracking began in 2010, the website regularly receives between 4,000-7,000 pageviews per month. Visitors come from more than 200 countries and territories, with the U.S., India, Kenya, South Africa, United Kingdom, Canada, Netherlands, Nigeria and Uganda topping the list. With an aim to increase use of evidence in policies and programs to reach women and girls, what difference has What Works made? Has evidence from it informed the policies and programs of donors? Has it informed national policies and plans? Has it contributed to the information used by civil society organizations? Has access to the website been sufficient or were other activities needed to enhance the use of the evidence from the website? This paper reviews the evidence for how the What Works knowledge translation platform has made a difference in the global AIDS response. Lessons learned from this effort may assist others who are working to create cultures of evidence. 2 CREATING A CULTURE OF EVIDENCE USE Methodology Data in this paper come from in-depth interviews with stakeholders and review of policy and plan documents. Using a snowball method, 36 in-depth interviews with representatives from key donors, namely the Global Fund for AIDS, TB and Malaria (GFATM); the President’s Emergency Plan for AIDS Relief (PEPFAR); UN agencies; and with key national policymakers, programmers, and representatives of civil society to assess if and how they used What Works and whether this resulted in any changes in donor policies, National Strategic Plans (NSPs) or in Concept Notes (CN) submitted to GFATM or additional outcomes at more local levels. NSPs and CNs from selected countries where interviewees noted the use of the website were reviewed to document if any of the key interventions from the website were included. All respondents identified in the paper agreed to be identified. Development of What Works for Women and Girls: Evidence for HIV/AIDS Interventions What became the website started out as a review and compilation of the evidence for women and girl-specific interventions that could inform country-level programming. This review was commissioned in 2008 by the Open Society Foundations’ (OSF) Public Health Program in response to skepticism by the then-Executive Director of the Global Fund to Fight AIDS, TB and Malaria (GFATM) that there was enough evidence on women- and girl-oriented HIV interventions to warrant programming. It built on earlier work to assess what works across a range of health topics (Hardee et al,. 2004; Wood et al., 2007). The review of evidence (Gay et al., 2010) and the subsequent development and launch of the website www.whatworksforwomen.org was created to translate scientific evidence from the peer reviewed literature for use by policymakers, program planners and donors developing policies and programs; by implementers carrying out programs; and also by civil society advocating for gender transformative programming. What Works is a widely available tool, rather than a static review document, and as such, the website promotes evidence-informed gender-responsive interventions for all areas of HIV. Platform Design The website, now owned and maintained by What Works Association, was created as a knowledge platform to provide a freely available, easy-to-use ‘one- stop shop’ of programming for women and girls with demonstrably successful results. Substantial thought went into the design and development of the platform to make it cleaner and more user-friendly than existing knowledge platforms. The platform was designed to maximize ease of use for those who work in the HIV field, but may not have a deep level BOX 1 Home page of www.whatworks.org, including a list of strategies covered THE EVIDENCE PROJECT 3 of expertise in every aspect of the HIV literature. A clean homepage design with clearly visible main topic sections provides the user with a big-picture perspective of what information is contained within the site. In order to avoid excessive clutter on the homepage, a hoverable ‘All Strategies’ tab at the top of the home page allows users to proceed directly to the topic area of interest. From the homepage, users can also proceed directly to the Methodology, Acknowledgements and References sections, as well as easily identify the latest updates in the News feature. Notably missing from the homepage are institutional logos and brand markings. While funding and institutional affiliations can be found under the ‘About Us’ tab, they were intentionally left off the homepage to provide a streamlined appearance, and more importantly, to avoid an affiliation with any one institution. As of 2013, What Works Association, an independent non-profit organization established by the original authors, owns the platform. Funding for updates has come from a variety of entities, including Open Society Foundations, the U.S. Agency for International Development (USAID), PEPFAR, the United Nations Development Programme (UNDP), the Joint United Nations Programme on HIV/AIDS (UNAIDS), and ViiV Healthcare. The main content of the web-based platform revolves around the numerous topics related to women and girls and HIV. Whereas other knowledge platforms serve as clearinghouses for information by topic, the What Works platform was designed to provide a more nuanced and analytic approach. Each main topic (e.g., Treatment) and sub- topics (e.g., Provision and Access, Adherence and Support, Staying Healthy and Reducing Transmission) include an Overview tab. Overviews provide a basic understanding of the main epidemiological statistics, themes, controversies and gender-related issues for users that may not be well-versed in that topic. The descriptive, bold section titles and pull-quotes enable a reader to quickly skim the narrative and hone in on areas of interest. The user can also hover over citations to see the full reference for further information. The Evidence tab lists the interventions and outcomes in plain, non-technical English. As the methodology for the platform content notes, research published in peer-reviewed publications and study reports with clear and transparent data on the effectiveness of various interventions for women and girls, as well as program and policy initiatives that can be implemented to reduce prevalence and incidence of HIV and AIDS and meet the needs of those living with HIV in low and middle income countries was reviewed. Interventions were NOT chosen a priori with a search for supporting evidence; rather, interventions emerged from the literature and were categorized by strength of evidence and geographic spread. This approach reduced potential bias associated with pre-selecting interventions Studies that support the intervention are listed in order of the strength of the evidence and further by date. Each study description has a standard format; listing the country where the study took place, the year in which the study was conducted, the methodology of the study (such as a quasi-experimental study), the sample size and the key outcome that supports the intervention as well as any key parts to the intervention that someone in a Ministry of Health or other relevant organization might want to know, followed by the Gray Scale rating of the strength of the evidence (described below) and the reference (See Box 2). In order to avoid duplication, studies are listed only once throughout the platform, with hyperlinked cross- referencing to other sections where relevant. Finally, key gaps in programming and research that emerged from the literature on the topic are listed in a separate Gaps tab to provide a guide to areas with documented needs. The platform is arguably the most comprehensive evidence base on women and HIV in existence, with an extensive reference section of more than 6,000 citations. The website is also searchable by key word to quickly focus on areas of interest and includes a Downloads page where each section is found in PDF form along with short briefs on special topics. Downloadable PDFs allow users in countries with limited internet access to print and read the 4 CREATING A CULTURE OF EVIDENCE USE content without the need for constant internet connectivity. While the website is in English, some downloadable material is available in French, Spanish and KiSwahili. A password- protected administrative page allows staff to update the platform content without the need for computer programming expertise. Lastly, the website uses topic-specific meta-tags for search engine optimization to allow the site to be more easily found through organic web searches such as Google and Bing. Scientific Rigor and Technical Accuracy Scientific rigor and technical accuracy is paramount to an effective knowledge translation platform. To ensure rigor and technical accuracy, all content on the website is reviewed by content experts: (the full list of reviewers is available under Acknowledgements on the homepage.) An in-person expert review meeting was organized in 2010 prior to the original launch of the platform, with experts in each topic area reviewing every section to ensure technical accuracy and provide comments, revisions and suggestions. A follow up methodology meeting was held in 2011 to ensure the scientific rigor of What Works and its usefulness for programming, including PEPFAR programming. Given the breadth of interventions related to HIV and AIDS, which range from structural interventions to clinical treatment, no one system for rating evidence is perfect. Experts endorsed the use of the Gray Scale, originally identified for use in assessing evidence-based medicine for the Cochrane Collection (Gray, 1997), and expanded to include evidence-based health care (Gray, 2009). The inclusion criteria for evidence included in What Works is broader than the GRADE system used by WHO, which privileges evidence from randomized control trials as strong and other evidence as weak. Table 1 shows the Gray Scale used in What Works. What Works incorporates two other dimensions in its methodology: the depth of evidence (how many studies support the intervention), and the breadth of evidence (how many countries contribute evidence to support the intervention). This is important for LMICs because interventions that “work” or are “promising” in certain contexts may not be appropriate or feasible in others. Platform Dissemination and Outreach Significant dissemination and outreach was undertaken to advance the use of the platform. The What Works team employed a multitude of strategies to promote the knowledge platform to inform evidence-based, gender- responsive programming and policymaking including electronic and physical dissemination; meetings with individuals; group presentations; conference posters, presentations and trainings; publishing of peer-reviewed papers, tying the platform to complementary organizations’ efforts and providing direct in-country training and technical assistance. Each of these is briefly discussed below. BOX 2 Intervention Strategies THE EVIDENCE PROJECT 5 Targeted Electronic and Physical Dissemination The website was officially launched just before the 2010 International AIDS Conference (IAC) in Vienna, Austria with a direct email blast to more than 1,000 targeted users as well as to multiple key HIV and reproductive health listserves. In addition, in order to make the evidence base more broadly available to those in areas with poor internet connectivity – as is the case in rural areas of many low and middle income countries, portable flash drives containing a static copy of the knowledge platform were widely distributed at the conference. Similarly, flash drives of the full platform update in 2012 were distributed at the International AIDS Conference in Washington, DC, along with targeted email blasts of the ongoing updates and cross-listing on sites such as the Communication Initiative and the AIDS Portal. Additional dissemination resulted from the What Works feature in the Health and Human Rights Resource Guide (FXB, 2013) published by the in the FXB Center at Harvard University. Individual and Group Presentations, including Conferences The What Works team has participated in the 2010-2016 IACs, disseminating the work through training in satellite sessions and poster presentations. The What Works team also made a number of presentations on material from the website for government and international audiences. Several examples include presentations on: women who inject drugs at a PEPFAR meeting in Ukraine in 2011; gender considerations along the treatment cascade and treatment for pregnant women living with HIV at the Office of the Global AIDS Coordinator (OGAC) in Washington in 2015. Presentations were given between 2010 and 2016 to the International Family Planning Conference in Dakar, Senegal in 2011; to the Population Association of America (PAA) in 2011; at a Congressional briefing in 2011; to the American Public Health Association (APHA) in 2011 and to UN Women in 2011. Type Strength of Evidence I Strong evidence from at least one systematic review of multiple well designed, randomized controlled trials. II Strong evidence from at least one properly designed, randomized controlled trial of appropriate size. IIIa Evidence from well-designed trials/studies without randomization that include a control group (e.g. quasi-experimental, matched case-control studies, pre-post with control group) IIIb Evidence from well-designed trials/studies without randomization that do not include a control group (e.g. single group pre-post, cohort, time series/interrupted time series) IV Evidence from well-designed, non-experimental studies from more than one center or research group. V Opinions of respected authorities, based on clinical evidence, descriptive studies or reports of expert committees. Note: Gray includes five types of evidence (Gray, 1997). For What Works, level III has been subdivided to differentiate between studies and evaluations whose design includes control groups (IIIa) and those that do not (IIIb). Qualitative studies can fall in both levels IV and V, depending on number of study participants among other factors. For more detail about these types of studies and their strengths and weaknesses (Gray, 2009). Also see the methodology section on www.whatworksforwomen.org for more information on development and use of the Modified Gray Scale. TABLE 1 Modified Gray Scale of the Strength of Evidence 6 CREATING A CULTURE OF EVIDENCE USE To further disseminate evidence on programming for women and girls, the What Works team have carried out a series of webinars from 2010 to 2016 with nearly 100 civil society organizations on topics ranging from mainstreaming adolescent needs in HIV responses to using What Works as a resource in GFATM CN development. Peer-Reviewed Papers and Other Publications To continue to create a culture of evidence use, the What Works authors drew from the evidence on the platform for peer-reviewed published papers on which HIV programs work for adolescent girls (Hardee et al., 2014a); the evidence for social and structural approaches to the HIV responses (Hardee et al., 2014b); and what works to meet the sexual and reproductive health needs of women living with HIV (Gay et al., 2011); and additional costing and cost-effectiveness research on gender and HIV with UNAIDS and the London School of Hygiene and Tropical Medicine (LSHTM) (Remme et al., 2014). Additionally, several topic areas, including gender-based violence, treatment, and scaling up adolescent programming were condensed into 4-6 page topical summaries for wider dissemination at conferences and meetings. Complementary Efforts The What Works team partnered with numerous organizations, including the ATHENA Network and Women4GlobalFund, to jointly advance the use of evidence in gender-responsive HIV programming. What Works became part of a comprehensive tool set for creating a gendered approach to the HIV response that included UNAIDS’ Gender Assessment Tool as well as UNDP’s On Course: Mainstreaming Gender into National HIV Strategies and Plans – A Roadmap. Training and Technical Assistance What Works team members provided orientation and/or training to a range of stakeholders, starting with GFATM staff and UNAIDS staff in 2010. In 2012, What Works team members presented the knowledge platform to the Kenyan National AIDS Control Council with participation from government officials and key civil society stakeholders. In 2013, a member of the What Works team briefed the PEPFAR/USG team in Kenya as well. What Works later partnered with UNAIDS to train a team of global consultants on how to use the website to inform the newly developed UNAIDS’ Gender Assessment Tool for use in adding gender-responsive components to NSPs and GFATM CNs. Two trainings on the use of the What Works platform took place in 2013 for almost 90 participants from more than 20 countries. Members of the What Works team later traveled to eight countries to provide in-person technical assistance to those undergoing the gender assessment process to strengthen gender- responsive national policies and programs. (Outcomes of this work are described in more detail below). Also in 2013, What Works team members were contracted by UNDP to orient Ministry of Health, AIDS Commissions and civil society organizations in Kenya and Uganda to both What Works and the UNDP gender mainstreaming checklist (UNDP, 2014) as well as in Malawi to develop and lead a capacity-building workshop for government and civil society stakeholders on using the evidence base to mainstream gender in national policies. The dissemination and outreach efforts described above have all aimed to enhance a culture of evidence use through several of the components noted by Hardee and Wright (2015). In addition to making research directly available through the knowledge platform itself, outreach efforts strengthened the capacity of those seeking to use evidence in their work, used intermediaries between researchers and decision-makers, and created better packaging and communication of findings. THE EVIDENCE PROJECT 7 Results What has come of these efforts? What difference has the platform and related dissemination and outreach activities made in advancing the use of evidence in HIV programming for women and girls? Interviews with key individuals and a review of 55 national policies and programming documents have found that What Works has had a substantial impact in promoting the use of evidence to effect positive change in HIV interventions for women and girls across donors and in a number of countries. Creating a culture of evidence has required building a unique knowledge platform, providing technical assistance and publicizing the availability of the evidence. Additionally, the focus of donors, specifically PEPFAR and Global Fund, on evidence-informed and gender- responsive interventions to qualify for funding, was extremely important. The UNAIDS’ Gender Assessment Tool facilitated greater use of evidence-informed, gender-responsive policies, including the use of the What Works platform. Effect on Donor Policies and Programs The What Works knowledge platform has served as a key resource for donor and aid organizations including the Global Fund, UNAIDS and PEPFAR. Global Fund to Fight AIDS, TB, Malaria (GFATM) Since the What Works platform grew out of the need to provide the GFATM with evidence-informed HIV interventions for women and girls, What Works has remained high on the radar of the GFATM gender team. The GFATM Senior Technical Advisor for Gender noted in her interview that she used the website in 2016 to inform gendered aspects of GFATM’s strategic framework for 2017-2022. In providing support to countries and civil society, the gender team has referred people to the What Works website as a useful resource in building evidence based cases for women and girls in CNs. The Information Note guiding implementers on gender equality: “Addressing gender inequalities and strengthening resources for women and girls” (GFATM, 2014) was co-authored by two members of the What Works team. A recent review commissioned by GFATM found that improved translation of gender equality was needed not just in policies, but also in programs and budgets (Middleton-Lee, 2016) and efforts by GFATM are currently underway to do so. UNAIDS The What Works team oriented the gender team at UNAIDS to the platform to help meet the needs of countries submitting CNs to the GFATM to address gender. UNAIDS developed a Gender Assessment Tool (GAT) to assist countries undertake a gender assessment to inform their NSPs and CNs (UNAIDS, 2014). When first piloted in 2013, then Chief of Gender Equality and Diversity Division at UNAIDS, Dr. Jantine Jacobi, noted that an early version of UNAIDS’ Gender Assessment Tool, prior to the engagement of What Works, generated long lists of problems without solutions. Dr. Jacobi had first learned about www.whatworksforwomen.org in the 2010 presentation at UNAIDS and said in her 2016 interview that she realized that What Works could contribute to the GAT by providing the evidence for gender transformative interventions. The platform was listed as a key resource in the GAT, and, as noted above, UNAIDS hired the What Works Association to provide in person or distance technical assistance in eight countries and to train more than thirty gender consultants on how to use the platform within the GAT process to ensure that evidence on gender responsive interventions for women and girls informed NSPs and GFATM CNs. 8 CREATING A CULTURE OF EVIDENCE USE PEPFAR PEPFAR representatives from the OGAC and from the USAID Mission in South Africa were invited to the 2010 review meeting in Cape Town, South Africa and PEPFAR began supporting What Works through the Health Policy Project led by the Futures Group (now the Palladium Group) and later through the Evidence Project led by the Population Council. What Works was used to develop PEPFAR’s Gender Strategy (PEPFAR, 2013), according to Dr. Daniela Ligiero, the Senior Gender Advisor at OGAC from 2010-2014, and she and Dr. Nina Hasen, Senior Technical Advisor for HIV Prevention at OGAC from 2009 to 2015 (who had attended the 2010 review meeting) both indicated that they used the resource to develop guidance on programming for women and girls for Country Operational Plans (COPs). In 2013, USAID, with PEPFAR funding, requested the What Works team to draw together all the evidence related to adolescent girls from the website, and to organize a meeting with key HIV, adolescents and gender experts plus members of the PEPFAR Gender Technical Working Group. The evidence from the website that was provided as a background paper for the expert meeting was then condensed into a published brief (Croce-Galis et al., 2014.). Dr. Ligiero noted in a 2016 interview that PEPFAR had used What Works, including articles, papers, meeting reports and reviews in developing the DREAMS Initiative – a partnership to reduce HIV infections among adolescent girls and young women in 10 sub-Saharan African countries. PEPFAR has further promoted attention to gender as part of the HIV programming response over the past several years. The updated PEPFAR Gender Strategy and COP guidance provides information on technical priorities and types of interventions that country teams should employ. Furthermore, for COP16, PEPFAR required a gender analysis be conducted and used to inform its country operational planning. Reporting through budget attributions, programmatic indicators, and narratives assist in tracking gender integration across PEPFAR’s global and country- specific interventions. Effect on Country NSPs and GFATM CNs The What Works platform had a notable effect on National Strategic Plans and GFATM CNs. This effect was largely due to the in-country technical assistance that What Works team members provided to country groups, especially those carrying out the gender assessment process as part of their GFATM CNs and NSP development. In some cases, the What Works team provided direct technical assistance and in other cases, UNAIDS consultants who were trained used the website for their UNAIDS Gender Assessments. Interviews and reviews of plan documents found that evidence from the What Works platform was found in NSPs and/or CNs for 14 countries that were developed between 2013 and 2016. This includes nine countries in Africa (Burundi, DRC, Djibouti, Gabon, Kenya, Malawi, Nigeria, Rwanda, and Uganda), two countries in the Caribbean (Dominican Republic, and Haiti) and three countries in Asia and the Near East (Malaysia, Myanmar, and Tajikistan). In some countries, evidence from the What Works website was only found in the UNAIDS Gender Assessments (GAs) – such as Lesotho, Egypt, Senegal, Zambia, Paraguay and Honduras. In other countries, evidence was found in the GA plus the NSP or GFATM CN (see Table 2). While inclusion in national strategies and GFATM CNs notes alone doesn’t guarantee implementation, it is an important first step in the process. Five of the 15 countries are highlighted here as examples of how the What Works knowledge platform, along with technical assistance provided for its use, prompted greater attention to and inclusion of evidence-informed HIV interventions for women and girls. THE EVIDENCE PROJECT 9 Malawi UNAIDS country office staff and national consultants attended the 2013 UNAIDS workshop introducing What Works as part of the Gender Assessment process. The capacity-building workshop for government and civil society led by a What Works team member that occurred in Malawi later that year was co-organized with a representative from UNDP who subsequently became a member of the core team in the Gender Assessment process. The What Works team member also arranged for the national consultant who would be carrying the Gender Assessment process to present that process at the Malawi workshop for government and civil society. This process led to What Works being cited as a “major resource” in the development of the national strategic plan as well as a resource for the Gender Implementation Plan of 2015. Representatives of women’s groups, some trained by a What Works team member, were engaged in the development of CN. The Gender Implementation plan noted the need to “focus on evidence based interventions for maximum impact” (Malawi, 2016). The Gender Assessments, the NSP and the CN all note the importance of women and girls as a key vulnerable population. Besides flagging the importance of evidence and women as a key vulnerable group, key recommendations from What Works incorporated into the NSP and CN included, among others, addressing gender based violence (GBV) within HIV services and provision of pre-exposure prophylaxis (PEP); integration of sexual and reproductive health (SRH) and HIV; peer education for sex workers; cash transfers; sex education; and scaling up both male and female condom use. Kenya While no UNAIDS Gender Assessment was conducted in Kenya, Dr. Nduku Kilonzo, Director of the National AIDS Control Council, as well as Allan Maleche, Executive Director, KELIN and Wanjiru Mukoma, Executive Director, LVCT Health, both members of the writing committee for the NSP, said in 2016 that they had used the What Works website as input to design the Kenya NSP (2014) and GFATM CN (Kenya, 2015b). Table 3 highlights the incorporation in the 2015 Kenya Plan on HIV and Adolescents (Kenya, 2015a) of some of the evidence- based interventions included in What Works to indicate the similarity of wording even though the resource was not cited directly in the Plan. The Kenyan CN noted that “the country will … continue to strengthen its rights and evidence-based approaches…through more gender responsive programming, advocacy and policy… (Kenya, 2015b, p. 20). Some of the evidence-based interventions from the What Works website that are also included in the NSP and/ or the GFATM CN were integration of HIV and family planning services; peer support groups for women living with HIV and pregnant women living with HIV; inclusion of partners of people who inject drugs; keeping girls in school; community base participatory learning approaches to create more gender equitable relationships; cash transfer programs to keep girls in school; peer education for sex workers; and PEP) (Kenya, 2014, Kenya, 2015b). Myanmar A What Works team member contributed to the 2013 gender assessment. Evidence-based interventions, as well as gaps, that were highlighted in the GA (Myanmar, 2013) were also noted in the evaluation of the 2011-2016 NSP on HIV/AIDS (Myanmar, 2015); the 2016-2020 NSP (Myanmar, 2016a) and the GFATM CN (Myanmar, 2016b). While What Works was only cited directly in the GA, the evidence and gaps contained in What Works were reflected in the NSP and CN. For example, a new focus on female partners of men who inject drugs was added to the 2016-2020 NSP (Myanmar, 2016a, p. 28), which was based on gaps listed in What Works and noted in the GA. The GFATM CN noted that there is overlap between key populations, including sex workers and people who inject drugs, and their partners (Myanmar, 2016b, p. 9). The CN also states that “all priority interventions through 10 CREATING A CULTURE OF EVIDENCE USE UNAIDS Gender Assessment National Strategic Plan Concept Note to GFATM AFRICA B u r u n d i November 2013 (summary only available for review) Integrate HIV and sexual and reproductive health • Provide increased information and services to sex workers • Integrate sex education into school curricula GBV services to reduce vulnerability to HIV Not available for review January 2015 • Integration of reproductive health with maternal health to reduce maternal mortality due to HIV • Increase support for peer educators as part of health system for mothers, male partners and people living with HIV to support adherence and for MSM and sex workers • Address Gender Based Violence (GBV) to reduce vulnerability to HIV • Mentor mothers D j i b o u t i July 2013 • Reorient prevention efforts and funding toward prevention for sex workers, along with peer education • Rapidly expand prevention of mother-to-child transmission (for the benefit of both infants and mothers’ health) • Train health workers on the law against discrimination of people living with HIV and training for health workers to not discriminate • Disaggregate data by age and sex • Need to address the high rates of gender based violence by conducting community based conversations Not available for review. Last NSP found 2005. January 2015 • Strengthen prevention of HIV among key populations and support peer education by sex workers • Eliminate mother to child transmission, including reduction of mortality among women • Notes the denial of rights to people living with HIV; Communities will be trained on human rights and gender • Epidemiological data will be analyzed based on age and sex TABLE 2 Use of Evidence from What Works in UNAIDS Gender Assessments (GA), National Strategic Plans (NSP) or other National Policy Documents and/or Concept Notes (CN) Submitted to the Global Fund for AIDS, TB and Malaria (GFATM) For each country, text in bold is what has been taken from What Works that is in Gender Assessments, and either NSPs, CNs or both. 10 CREATING A CULTURE OF EVIDENCE USE THE EVIDENCE PROJECT 11 UNAIDS Gender Assessment National Strategic Plan Concept Note to GFATM D R C 2013 • Services for GBV survivors • Increased micro-enterprise for women • Need for community based training on gender norms • Epidemiological surveillance that disaggregates by sex • Sex education • Provider training to reduce stigma and discrimination • Use of Stepping Stones • Cash transfers • Training to increase women’s autonomy • Girls’ education Not available for review. March 2014 • Focus on reducing gender inequality and GBV • Linking maternal and reproductive health services with services for GBV • Income generating activities for women • Strengthen prevention of HIV among key populations and support peer education by sex workers • Eliminate mother to child transmission • Integration of family planning to prevent unintended pregnancies • Screening for cervical cancer • Treatment for syphilis among pregnant women • Procurement of both male and female condoms G a b o n 2013 • Promote women’s gender equality and autonomy • Women as a key population • Increase access to sex education • Peer education for sex workers • Increase access to income for women • Train women on their rights in terms of GBV • Increase contraceptive options for women living with HIV • Address multiple partnerships and transgenerational sex and increase knowledge of HIV and prevention by young girls June 2014 • Promote gender and women’s autonomy • Women listed as a key population • Sex education • Peer education for sex workers • Access to income generating activities for women • Training in gender with knowledge of rights when faced with GBV • Provision of contraception within programs to prevent vertical transmission • Addressed early sexual debut, multiple partnerships, lack of knowledge by youth on prevention modalities, and transgenerational sex • Eliminate mother to child transmission • Increased targeted assistance for Orphans and Vulnerable Children (OVC) January 2014 • Focused on TB 12 CREATING A CULTURE OF EVIDENCE USE UNAIDS Gender Assessment National Strategic Plan Concept Note to GFATM K e n y a GA not conducted in Kenya but WW was used directly in the NSP and CN (see text) 2015 • Implement GBV prevention and response • Integration of HIV and family planning services • Reduce stigma and discrimination • Adherence support • Peer support groups for women living with HIV • Inclusion of partners of people who inject drugs • Keeping girls in school • Pre-exposure prophylaxis (PEP) • Strengthen prevention of HIV among key populations and support peer education by sex workers • Community based participatory learning approaches to create more gender equitable relationships • Eliminate mother to child transmission • Cash transfer programs to keep girls in school 2015 • Reduce GBV • Integration of HIV and family planning services • Reduce stigma and discrimination • Peer support groups can increase adherence • Inclusion of partners of people who inject drugs • Keeping girls in school • Pre-exposure prophylaxis (PEP) 12 CREATING A CULTURE OF EVIDENCE USE THE EVIDENCE PROJECT 13 UNAIDS Gender Assessment National Strategic Plan Concept Note to GFATM M a l a w i July 2014 • Importance of women and girls as key vulnerable group • Address GBV and integrate GBV in HIV services • Integrate SRH in HIV programs • Disaggregate data by age and sex • Address the needs of sex workers, including scaling up economic empowerment • Stepping Stones as a model program • Promote both female and male condom programming, also for youth • Promote sex education • Develop a strategy for women and girls in NSP • Risks of nonvaginal sex • Provide condoms in prisons • Provide an enabling environment • Gap of transactional sex and child marriage 2014 • Key vulnerable population are adolescent girls and young women, and sex workers • GBV reduction programs and provision of PEP • Integrate of SRH and HIV • Disaggregate data by sex • Peer education for sex workers • Cash transfers • Scale up both female and male condom uptake • Importance of human rights and gender • Informed reproductive choices • Men trained to provide care • Research on how to increase cervical cancer screening among women living with HIV 2015 • Adolescent girls and women as key population • Link GBV survivors with PEP • Integrate SRH and HIV • Need for sex disaggregated data • Programs for sex workers and clients of sex workers, with clinics that are friendly to sex workers and peer education • Stepping Stones as a model program • Scale up of both female and male condoms • Sex education • Cash transfers • Train teachers on sex education • GAP: difficulties for adolescents accessing SRH and HIV prevention services, cites GA • Address gender norms • Reduce stigma and discrimination • Reduce GBV • Programming on GBV and gender norms; vulnerability of women and girls • Promote early infant male circumcision • Mother to Mother programs • Counseling for men and their female partners on VMMC • Training on human rights • Importance of DREAMS/PEPFAR • Operational research on gender and human rights 14 CREATING A CULTURE OF EVIDENCE USE UNAIDS Gender Assessment National Strategic Plan Concept Note to GFATM N i g e r i a December 2013 • Need for sex education • Noted that women and girls are disproportionally affected by HIV and that data on HIV treatment are not disaggregated by sex • Need for GBV services and PEP services • Importance of girls’ education • Need for increased employment opportunities • Support for OVC • Cash transfers • The need for HIV prevention, treatment and care for those past reproductive age Not available for review. January 2015 • Sex education • Noted that the provision of ART would be gender sensitive • Mentions gender and cites to GA regarding gender related gaps, such as child marriage, GBV, and harmful gender norms, among others • Notably, no interventions for GBV were included in the CN, despite GBV being listed as a gap. • Integration of sexual and reproductive health into HIV programs • Programming for CSOs to monitor and report on gender issues R w a n d a 2013 • Providing GBV services • Limited girls’ education as a risk for HIV acquisition and address structural barriers that reduce access by girls’ education • Promote gender equality as a key goal to reduce vulnerabilities of women and girls • Strengthen the capacity of sex workers to participate in the HIV response • Reduce HIV- attributable deaths among women • Community based training on gender norms • Integration of family planning with services for cervical cancer within HIV programs • Scaling up of sex education • Scaling up PEP Not available for review. March 2014 • Providing GBV services • Girls education • Need to address gender inequalities which prevent young girls from negotiating safer sex • Strengthen prevention of HIV among key populations and support peer education by sex workers • Eliminate mother to child transmission, including reduction of mortality among women • Proposes to measure access to treatment for HIV positive pregnant women, rather than just measuring infant outcomes • Educational support for OVC 14 CREATING A CULTURE OF EVIDENCE USE THE EVIDENCE PROJECT 15 UNAIDS Gender Assessment National Strategic Plan Concept Note to GFATM U g a n d a In lieu of a GA, the Uganda National Gender and HIV Scorecard, 2014 was issued and noted: • The need for strengthening the enabling environment for women and girls • Need to include gender • Need to integrate SRH and HIV • Need to integrate HIV and GBV • Need to promote “evidence based” (p. 27) gender sensitive approaches • Need for information on if VMMC increases risk of HIV acquisition for women • Gap of women as caregivers • Need multisectoral approaches 2015 • Guiding principles included “gender –based approach to programming”(p. 19) • Address gender norms • Integrate SRH with HIV • Address norms on GBV • Integrated GBV prevention into HIV programming • The NSP “emphasizes the need for scaling up evidence based effective interventions” (p. 9) • Scale up sex education • Reduce stigma and discrimination against women and girls • Scale up distribution of both female and male condoms October 2014 • Gender inequality as a key driver of HIV epidemic and the need to focus on gender (p. 23) • Address gender norms • Integrate SRH and HIV • Importance of GBV • Sex education • Need for syphilis in PMTCT • Importance of addressing needs of sex workers and their clients • Peer education for sex workers • Scale up condoms ASIA M a l a y s i a No Gender Assessment was conducted. Not available for review. January 2014 (consultant trained by What Works team provided information on it to the CN writing team) • The need to provide services for sexual partners of people who inject drugs • Training providers to reduce stigma and discrimination • Address sexual and reproductive health and rights for women living with HIV • Provide PEP for GBV • Community service delivery mechanisms for providing treatment 16 CREATING A CULTURE OF EVIDENCE USE UNAIDS Gender Assessment National Strategic Plan Concept Note to GFATM M y a n m a r December 2013 • Noted the need to focus on female partners of men who inject drugs • Noted that PMTCT interventions were not yet widely available • Need to integration of HIV, TB, maternal health and sexual and reproductive health • Need to disaggregate data by age and sex December 2016 • Noted that there is overlap between key population, including sex workers and people who inject drugs, and their partners • Focus on female partners of men who inject drugs June 2016 (revised by Myanmar at GFATM’s request to add more on gender and human rights) • Listed gender as a guiding principle • PMTCT services will be widely available • States that “all priority interventions through intensified outreach component will improve penetration of key population networks reaching those that have not been reached and those that may have been underserved such as women and young girls who are either partners of key populations or are themselves the key populations” • Enhanced integration of HIV, TB, maternal health and sexual and reproductive health • Disaggregate data by age and sex T a j i k i s t a n August 2013 • Focus on scaling up opioid substitution therapy (OST) and recognizing the needs of women who inject drugs • Need to provide OST within prisons • Conduct educational sessions for people who inject drugs and their partners and to expand services to include female injecting drug users as well as sexual partners of people who inject drugs • Included access to female condoms and sexual and reproductive health services and a focus on HIV prevention for sex workers • Address the links between HIV and GBV and to provide holistic services to GBV survivors, including PEP, which were previously separate • Promote gender equity • Enforce property rights for women • Invest in HIV positive women’s network Unable to review- NSP in Russian. October 2015 • Focus on scaling up opioid substitution therapy (OST), including for women who inject drugs • Provide OST within prisons • Conduct educational sessions for people who inject drugs and their partners and to expand services to include female injecting drug users as well as sexual partners of people who inject drugs • Included access to female condoms and sexual and reproductive health services and a focus on HIV prevention for sex workers • Address the links between HIV and GBV and to provide holistic services to GBV survivors, including PEP • Notes that: “for the first time the problem of women and girls, gender equality and related issues of HIV infection were included in the broad national agenda. 16 CREATING A CULTURE OF EVIDENCE USE THE EVIDENCE PROJECT 17 UNAIDS Gender Assessment National Strategic Plan Concept Note to GFATM THE CARIBBEAN D R 2014 Providing Post-exposure Prophylaxis (PEP) in cases of sexual violence • Need to disaggregate data by sex • Provide peer support for women living with HIV • Training providers concerning the rights of women living with HIV September 2014 • Provision of PEP for GBV survivors • Promotes gender equality • Noted gaps in GBV, inequitable gender norms, need to name women a key population due to the feminization of the epidemic, need for gender training, and need for health services structured to meet women’s needs • Notes the need to scale up sex education • Notes the need to train peer education for sex workers • Scaling up distribution of both male and female condoms • Community based training on gender and GBV February 2015 • Biomedical focus on treatment, with no focus on gender- responsive programming H a i t i April 2014 • Noted the need to scale up sexual education • Provide male and female condoms • Noted the need for peer education for sex workers • Noted the need to providing family planning options to women living with HIV to reduce unintended pregnancies May 2014 • Scaling up sexual education • Providing male and female condoms • Undertaking peer education for sex workers • Providing family planning options to women living with HIV March 2014 • Submitted before the GA was undertaken 18 CREATING A CULTURE OF EVIDENCE USE intensified outreach component will improve penetration of key population networks reaching those that have not been reached and those that may have been underserved such as women and young girls who are either partners of key populations or are themselves the key populations” (Myanmar, 2016b, p. 54). Another key issue addressed by the GA was that PMTCT interventions were not yet widely available (Myanmar, 2013, p. 6). In response, the 2016 GFATM CN stated that PMTCT services will be widely available (Myanmar, 2016b, p. 43). Finally, within the 2013 Myanmar GFATM CN, no mention was made of gender or women, whereas the 2016 GFATM CN listed gender as a guiding principle. While not mentioned in the 2016 CN, the GA led to the establishment of an ongoing Working Group on Gender, Human Rights and HIV, with ongoing training on gender and human rights, according to Ms. Yuki Takemoto, UNAIDS and Ms. Tina Boonto, UNAIDS. The GFATM Technical Review Panel (TRP) requested Myanmar to revise its 2016 CN with increased attention on programs on gender and human rights. Boonto also said that Myanmar now has “enhanced integration of HIV, TB, maternal health and sexual and reproductive health, which was a clear recommendation of the UNAIDS gender assessment that drew on What Works.” She added that another key recommendation adopted based on the GA was to disaggregate data by age and sex. Myo Thet Oo, a member of a civil society organization in a 2016 interview claimed that: “What Works is how we got a budget allocation for gender based violence into our Myanmar HIV National Strategic Plan,” although it was not possible to independently verify this claim. Tajikistan Members of the What Works team participated in 2013 GA and the evidence base from What Works was cited in the recommendations. In agreement with evidence from What Works and the recommendations from GA, the 2015 CN submitted to the GFATM included focus on scaling up opioid substitution therapy (OST) and recognizing the needs of women who inject drugs, “with a need to strengthen gender sensitive harm reduction” (p.13). Significantly, the CN adopted the recommendation from the GA (Tajikistan, 2013) to conduct educational sessions for people who inject drugs and their partners and to expand services to include female injecting drug users as well as sexual partners of people who inject drugs. Also included from the GA recommendation was the need to provide OST within prisons. The GFATM CN notes that: “…for the first time the problem of women and girls, gender equality and related issues of HIV infection were included in the broad national agenda…” (p. 16). Based on the GA, the CN also included access to female condoms and sexual and reproductive health services and a focus on HIV prevention for sex workers. The GA and evidence from What Works were used to address the links between HIV and GBV and to provide holistic services to GBV survivors (Tajikistan, 2015) (Gay, 2013). Use of What Works by Civil Society Civil society organizations have also used the website to advocate for attention to gender issues in the HIV response, or to use as a basis for their work plans or funding proposals. Examples from three civil society organizations (Silver Rose, KELIN, and UGANET) are given here. Civil Society Organizations (CSOs) learned about What Works through participation in UNAIDS GAs, as reviewers, and through UNDP site visits. Zhenya Maron stated that she used What Works to develop evidence-based interventions for the sex worker movement in Russia, Silver Rose, which has been funded by the Robert Carr Civil Society Networks Fund, Mama Cash and the Red Umbrella Fund. Silver Rose Movement is a civil society organization and was established in 2006 in St. Petersburg with 450 leaders of sex workers form 30 regions of Russia and is a leader in HIV prevention and service delivery. THE EVIDENCE PROJECT 19 Additionally, KELIN, a civil society organization in Kenya, used the website to write their successful DREAMS proposal, according to Allan Maleche, Executive Director, KELIN, in an interview in 2016. Their DREAMS project engages the justice system in order to secure the sexual reproductive health rights of adolescent and young women (KELIN, 2016). Dora Kiconco, Executive Director of UGANET reported using the What Works platform to create a gender “Scorecard” that reviewed the Uganda NSP and provided a gender “score” that showed large gaps in implementation of gender programming in the NSP (Uganda, 2014a). It was on the basis of this evidence that the GFATM awarded funds to Uganda for gender programming under the Health Systems Strengthening grant funds. Another Scorecard review is scheduled in the coming year and this time the Uganda AIDS Commission will be taking the lead, rather than a CSO (UGANET). In Kenya’s 2015 NSP From What Works Ensuring adolescents are retained in school and complete their education Increasing educational attainment can help reduce HIV risk among girls Sex and HIV education, prior to the onset of sexual activity may reduce stigma, delay first sex, increase HIV testing, condom use and reduce the number of sexual partners Sex and HIV education with certain characteristics prior to the onset of sexual activity may be effective in reducing stigma and preventing transmission of HIV by increasing age of first sex and for those who are sexually active, increasing condom use, testing, and reducing the number of sexual partners Training of teachers to teach age appropriate comprehensive sexuality and AIDS education and increases knowledge Training for teachers to conduct age-appropriate participatory sexuality and AIDS education can improve students’ knowledge and skills Mass media and social marketing campaigns can change risky behavior Mass media and social marketing are modestly effective in persuading both female and male adolescents to change risky behaviors Increasing income generating opportunities can reduce sexually risky behavior Increased employment opportunities, microfinance, or small scale income generating activities can reduce risky behavior – particularly among young women TABLE 3 Link between Interventions for Women and Girls in Kenya. 2015 a. Kenya’s Fast- track Plan to End HIV and AIDS among Adolescents and Young People and the 2012 Interventions Highlighted in the 2012 Version of What Works Note: The current version of What Works on the website has been updated since the 2015 NSP was being written; the wording in this table is from the earlier version of What Works so while the interventions have not changed, the current wording may be slightly different. 20 CREATING A CULTURE OF EVIDENCE USE Discussion While it is difficult to draw direct lines to policy change, this paper shows that access to evidence informed gender transformative interventions through the knowledge translation platform anchored by www.whatworksforwomen. org has been used in a number of donor policies and national governmental plans and requests for funding. This paper shows evidence that What Works was used in GAs in 16 countries, was used in NSP in seven countries, and was used in GFATM CNs in 11 countries. Additionally, civil society organizations have used the website to advocate for evidence-informed gender transformative policies within their national policies and/or used the evidence base to create programs that are grounded in evidence. Examples of civil society use of What Works are shown for three countries. Creating a user-friendly platform, accepted as a key resource by major donors, with technical assistance by either members of the What Works team or those trained by What Works contributed to a culture of evidence in HIV globally. Users of this resource have noted the utility of having a range of up-to-date and vetted evidence in one place that is easily accessed on the web. They have also noted that the evidence is packaged in a way that speaks to programs – that research is made accessible to a range of stakeholders for their purposes. Some comments include: What Works is a “usable tool, where everything you want is easily found” and you can “get what you want quickly,” and that “One hour spent on What Works saves me 5-6 hours of looking up research on my own.” No one resource or tool can be solely responsible for changing policies and programs. Numerous governments, organizations, advocacy groups have come together to prioritize gender-responsive HIV programming and What Works is but one part of that large effort. The context in which the What Works platform came together was one in which donors, developing countries and UN agencies were all working to come up with evidence-informed programming for HIV with the most coordinated global effort on a public health topic over the past two decades: the 3 Ones campaign, where all agreed to channel resources in places where there was one national strategy, one coordinating body and one monitoring and evaluation plan (UNAIDS, 2004). Each resource and tool fulfills a niche and What Works provides the critical evidence focused on women and gender-related HIV issues. Dr. Kilonzo, Executive Director of the National AIDS Council, Kenya stated that the What Works platform identifies gender transformative interventions, while WHO is more focused on biomedical solutions. A noted value of the website is that the evidence it contains covers all aspects of programming (prevention, care, and treatment, in addition to the enabling environment and health systems) rather than being based on narrow questions needed to undertake systematic reviews. The range of topics covered allows stakeholders to find evidence for the policy and program questions that are relevant for their country and context. By seeing the range of countries the data for an intervention comes from, stakeholders can see if the intervention has been implemented in their own country or countries similar to theirs. While some interventions – such as clean needle exchanges, increased access to HIV treatment – are biologically true no matter in what context, most interventions need to be adapted to meet local contexts. However, access by policymakers and programmers to the evidence base is needed so that they can decide what gender responsive programming could potentially affect the HIV epidemic in their contexts and how evidence can be adapted. Champions have been important for supporting, publicizing and using the resource, including donors who have recommended What Works to their grantees as a source of evidence-based programming. Funding has come from a range of donors, most notably the OSF, which saw the initial need for the evidence, the utility of making it available on the web and launching it at an IAC, and PEPFAR, which supported subsequent updates of the evidence to keep the resource current. UNAIDS and the GFATM have been major users of the resource and THE EVIDENCE PROJECT 21 have facilitated its use at the country level. Some consultants trained by the What Works team through UNAIDS or UNDP have used What Works in multiple countries. Content experts have supported the work by providing reviews of the material on the website. Communications staff at the Futures Group (currently the Palladium Group), Population Council, Population Reference Bureau, and USAID has contributed to developing and disseminating products and announcements about What Works. There are limitations to this type of resource, however. The main challenge is that the platform loses its timeliness if it isn’t kept up to date, which is a labor-intensive activity. In addition, one of the main benefits of the platform – that it is not attached to one donor – is also a drawback in that it has been difficult to attract donors beyond piecemeal updates. Further, a number of requests have been fielded about translating the website into other languages besides English. And while some briefs have been translated into French, Spanish and KiSwahili, translating (and updating translations) has been beyond the scope of available resources. The platform has also been criticized for being skewed toward African countries, however, the evidence contained on the website reflects the fact that many of the studies have been conducted in Africa. Access to and use of the evidence is an important first step. But particularly in HIV, there are many gender-related issues that still, after more than thirty years, do not have a solid evidence base on which to program (see gaps in each section of www.whatworksforwomen.org). Respondents, including UNAIDS consultant Kibibi Thomas Mbwavi and Alice Welbourn of Salamander Trust (creator of the original Stepping Stones program), also noted the need for more tools and guidance for countries on how to adapt successful interventions to different country contexts, whilst ensuring the original program quality. Others, including Heather Doyle from the GFATM, described the need for accountability mechanisms so that strategies, policies and programs that have been included in NSPs or GFATM CNs are implemented. The experience with What Works illustrates the challenges of getting evidence into policies and programs at global and national levels. While access to evidence is necessary, it is not sufficient to transform the epidemic. Although the platform can stand alone, it is important to note that in many cases where direct outcomes of evidence use was found, use of the platform was supported by technical assistance, either directly by What Works team members or those who had been oriented to it. Further, Motoko Seko, former Technical Advisor, Gender at the GFATM from 2011 to June, 2016, reflected that the platform has a lot of potential and opportunities if there is political will to use it. Research on evidence use in decision-making shows that a range of factors affects health policy and program decision-making (Hardee and Wright, 2015). Cookson (2005) contends that scientific evidence is filtered through decision-makers’ beliefs about and values regarding an issue. Belief that there is evidence that gender programming makes a difference is important. Belief in adherence to evidence, rather than relying only on personal beliefs and values, is also an important factor in use of evidence. Clearly for some of the policy and program documents reviewed in this paper, beliefs and values took precedence over evidence. Yet, having the evidence base allowed CSOs to advocate for evidence-based programming and donors to request revised plans that adequately addressed gender and human rights. To be fair, it is also important to note that gender is one of many issues that have to be dealt with by decision-makers in the NSPs and CNs, as well as the PEPFAR COP process, so all relevant gender- responsive interventions could not necessarily be included. Still, HIV programming has suffered over the years with programming based on values rather than science, such as abstinence only sex education (IOM, 2001), which add to young women’s already high risk of HIV acquisition. It emerged from the analysis that not all uses of evidence from the website were attributed thorough citation to What Works. The point of the website is use of the evidence to promote gender-responsive programming rather than 22 CREATING A CULTURE OF EVIDENCE USE credit to What Works; however, lack of explicit reference to the resource gives the inaccurate impression of the breadth and depth of use. Assumptions that it is not being used may lead to reduced funding for the resource, which would lead to lack of availability of up-to-date and easily accessed information on gender-transformative approaches to address HIV/AIDS. BOX 3 The value of What Works • Publically accessible web-based one stop shop with the range of evidence on HIV/AIDS interventions for women and girls • Expert-reviewed evidence useful for policymakers, programmers, advocates and researchers • Focus on gender-responsive programming and doesn’t ignore men and boys • Puts the science in programmatic context and programmatic language • Presents the evidence for interventions, rather than promoting favored interventions • Inclusion criteria reflects programming and geographic spread of the evidence • The evidence covers all aspects of programming (prevention, care, and treatment, in addition to the enabling environment and health systems) • The utility of the website and evidence is augmented with training and technical assistance. Trained consultants use the resource in multiple countries THE EVIDENCE PROJECT 23 Limitations This analysis has limitations. The sample of 36 stakeholders interviewed was small and likely not representative of the universe of actual and potential users of What Works. Fifteen people did not respond to requests to be interviewed for this analysis. The website has been part of a global movement with many advocates to promote the use of evidence to create a gender transformative response. It is challenging to note what was solely due to What Works and what was due to the work of many other important groups and advocates, such as ICW, Women4Global Fund, Salamander Trust and ATHENA Network, among many others. The analysis was limited by language, which included review in English, French and Spanish. It was beyond the capacity of the authors to review documents in other languages, for example, the National Strategic Plan for Tajikistan that was written in Tajik and Russian. Web analytics may underestimate use since some users may download the chapters and use them without going back to the website. It is not possible to measure the effect of the website alone, compared to the website in addition to the training and TA on its use. Because it is a publically accessible website available at no cost, there may have been many more policies and programs that were influenced by evidence from the website that this analysis was unable to track. It was not possible to follow up with the people from 190 countries that have accessed the website, although web analytics do show which countries they are from and what topics were most accessed. Finally, getting gender-responsive interventions into policies and plans does not mean they will be implemented. Evaluation of country programs will be important to assess the impact of gender-responsive programming on HIV/AIDS outcomes. 24 CREATING A CULTURE OF EVIDENCE USE Conclusion This paper shows the value of a knowledge translation platform, www.whatworksforwomen.org and associated training and technical assistance, for providing the evidence to shape HIV/AIDS policies and programs for meeting the needs of women and girls. Interviews and review of documents has shown that What Works has played a role in shaping policies and programs in donor organizations and in selected country policies and submissions to the GFATM to request funding. Within certain countries, CSOs were able to advocate for certain gender transformative evidence based programs with their governments based on the evidence contained in the website. There is a clear need for a one-stop shop for evidence, supported by training and technical assistance to use the evidence to promote gender-responsive policies and programs to meet the continued prevention, care and treatment needs of millions of women and girls, in addition to men and boys, worldwide. The next step globally is to assess how and if evidence-informed gender transformative policies and programs have been effectively implemented, and if so, what difference has this made in the HIV epidemic? THE EVIDENCE PROJECT 25 References Burundi. Single Concept Note for Tuberculosis and HIV- Burundi. Geneva: Global Fund, 2015. Cookson R. Evidence-based policy making in health care: what it is and what it isn’t. J Health Serv Res Policy. 2005;10(2):118-21. doi: 10.1258/1355819053559083. Croce-Galis M, Hardee K, Gay J. Scaling up evidence-informed HIV prevention for adolescent girls and young women. Policy Brief. Washington DC: Health Policy Project, 2014. Democratic Republic of Congo. Gender Assessment. Analyse des résultats de l’évaluation basée sur le genre de la riposte nationale au VIH : des enjeux aux solutions proposées : cas de la RDC. 2013. Democratic Republic of Congo. Single TB and HIV Concept Note- Democratic Republic of the Congo. Geneva: Global Fund, 2014. DFID. (Department for International Development). What is the evidence on the impact of research on interna- tional development?: A DFID Literature Review. Version 1.1. Glasgow: UKAID Department for International Development, 2014. Dickson K, Tran N, Samuelson J, Njeuhmeli E, Cherutich P, Dick B, et al. Voluntary medical male circumcision: A framework analysis of policy and program implementation in eastern and southern Africa. PLoS Med. 2011;8(11):e1001133. Dominican Republic. NSP. Plan estrategio nacional para la respuesta a las ITS y al VIH- SIDA 2015- 2018. 2014. Dominican Republic. Standard Concept Note- Dominican Republic. Geneva: Global Fund, 2015. Eyben R. Uncovering the Politics of ‘Evidence’ and ‘Results’: A Framing Paper for Development Practitioners. Proceedings of the Politics of Evidence Conference. Brighton: 2013. FXB. Health and Human Rights Resource Guide. Cambridge: FXB Center for Health and Human Rights and Open Society Foundations, 2013. Gabon. Gender Assessment. Rapport d’evaluation de la prise en compte du genre dans la riposte nationale au VIH. 2013. Gabon. NSP. Plan stratégique national de lutte contre le VIH/Sida et les IST 2013-2017. Gabon. 2014a. Gabon. Concept Note Tuberculosis Gabon. Geneva: Global Fund, 2014b. Gay J. Trip Report to Tajikistan. 2013. Gay J, Hardee K, Croce- Galis M, Kowalski S. What works for women and girls: Evidence for HIV and AIDS inter- ventions. New York: Open Society Institute, 2010. Gay J, Hardee K, Croce-Galis M, Hall C. What Works to Meet the Sexual and Reproductive Health Needs of Women Living with HIV/AIDS? J Int AIDS Soc. 2011;14:56. GFATM. (Global Fund). Addressing gender inequalities and strengthening responses for women and girls: Infor- mation Note. 2014. Gray J. Evidence Based Health Care: How to Make Health Policy and Management Decisions. London: Churchill Livingstone, 1997. 26 CREATING A CULTURE OF EVIDENCE USE Gray J. Evidence-Based Health Care and Public Health: How to Make Decisions About Health Services and Pub- lic Health. 3rd Edition. Edinburgh: Churchill Livingston Elsevier, 2009. Haiti. Gender Assessment. Evaluation de l’intégration du genre dans la riposte nationale a VIH/SIDA en Haïti. 2014a. Haiti. NSP. Plan strategique national multisectoriel 2012- 2015 revise avec extension a 2018. 2014b. Haiti. Concept Note Haiti. Geneva: Global Fund, 2014c. Hardee K, Gay J, Croce- Galis M, Peltz A. Strengthening the Enabling Environment for Women and Girls: What is the Evidence in Social and Structural Approaches in the HIV Response? J Int AIDS Soc. 2014b;14:18619. Hardee K, Gay J, Croce-Galis M, Afari-Dwamena N. What HIV program work for adolescent girls? J Acquir Im- mune Defic Syndr. 2014a;66(Suppl 2):S176-S85. Hardee, K, M Strachan, A Bhuyan, B Herstad, C Wingfield, and F Young. 2004. Ending Inequity: Strengthening the HIV/AIDS Response for Women.” Paper prepared for USAID. Washington, DC: POLICY Project. Hardee K, Wright K. Expanding the Role of Evidence in Family Planning, Program, and Practice Decision-mak- ing. Working Paper. Washington, DC: Population Council, The Evidence Project, 2015. Heidari S, Eckert M, Kippax S, Karim Q, Sow P, Wainberg M. Time for gender mainstreaming in editorial policies. J Int AIDS Soc. 2011;14(11). IOM. (Institute of Medicine). National Academy of Sciences. No Time to Loose: Getting More from HIV Preven- tion. Washington, DC: National Academy Press, 2001. KELIN. KELIN participates in the Kisumu County Court Users Committee Meeting. 2016. Kenya. NSP. Kenya AIDS Strategic Framework 2014/2015-2018/2019. Nairobi: National AIDS Control Council, 2014. Kenya. Kenya’s Fast- track Plan to End HIV and AIDS among Adolescents and Young People. Nairobi: National AIDS Control Council, 2015a. Kenya. TB and HIV Concept Note. Global Fund, 2015b. Kim J. WHO ProTest Initiative in South Africa: A Utilization Framework Case Study. In: Department of Reproduc- tive Health and Research W, editor. Turning Research into Practice: Suggested Actions from Case-studies of Sexual and Reproductive Health. Geneva: WHO; 2006. p. 51-6. Lavis J, Oxman A, Lewin S, Fretheim A. Support tools for evidence-informed policymaking (STP) 3: Setting priori- ties for supporting evidence-informed policymaking. Health Res Policy Syst. 2009;7(Suppl 1):S3. Malawi. Gender Assessment. Gender Assessment of the Malawi National HIV Response. 2014. Malawi. NSP. National Strategic Plan for HIV 2015-2020. Lilongwe: National AIDS Commission, 2014. Malawi. TB and HIV Concept Note. Geneva: Global Fund, 2015. Malawi. Gender and HIV and AIDS Implementation Plan 2015-2020. Lilongwe, Malawi: Ministry of Gender, Children, Disability and Social Welfare, 2016. Malaysia. Global Fund. CCM Request to Access Funding- Malaysia. Geneva: Global Fund, 2014. THE EVIDENCE PROJECT 27 Middleton-Lee S. Rapid Review: Results, gaps and lessons from implementation of strategies/action plans on gender equality and key populations – The Global Fund to Fight AIDS, Tuberculosis and Malaria. GFATM, 2016. Myanmar. Gender Assessment. Gender Assessment of the National HIV Response in Myanmar. 2013. Myanmar. Technical Strategic Group on HIV and AIDS. 2015. Myanmar. NSP. National Strategic Plan on HIV and AIDS. Myanmar 2016- 2020. 2016a. Myanmar. Concept Note for Early Applicants. Myanmar- HIV. Global Fund, 2016b. Nigeria. Gender Assessment. Gender Assessment of the National Response to HIV/AIDS in Nigeria. Abaju: Na- tional Agency for the Control of AIDS, 2013. Nigeria. TB and HIV Concept Note- Nigeria. Geneva: Global Fund, 2015. PEPFAR. Updated Gender Strategy. PEPFAR, 2013. Piot P. AIDS between Science and Politics. New York: Columbia University Press; 2015. Remme M, Siapka M, Vassall A, Heise L, Jacobi J, Ahumada C, et al. The cost and cost-effectiveness of gen- der-responsive interventions for HIV: a systematic review. J Int AIDS Soc. 2014;17:19228. Rwanda. Gender Assessment. Gender Assessment of Rwanda’s National HIV Response. 2013. Rwanda. TB and HIV Concept Note- Rwanda. Geneva: Global Fund, 2014. Spicer N, Bhattacharya D, Dimka R, Fanta F, Mangham- Jeffries L, Schellenberg J. ’Scaling-up is a Craft Not a Science’:Catalysing Scale-up of Health Innovations in Ethiopia, India and Nigeria. Soc Sci Med. 2014;121:30 - 8. Tajikistan. Gender Assessment. From Gender Assessment to Recommendations: Addressing the HIV Response in Tajikistan. 2013. Tajikistan. Concept Note- Tajikistan. Geneva: Global Fund, 2015. Uganda. TB and HIV Concept Note. Geneva: Global Fund, 2014. Uganda. The Uganda National Gender and HIV Scorecard 2014. 2014a. Uganda. National HIV and AIDS Strategic Plan 2015/2016 - 2019/2020. 2015b. UNAIDS. (Gender Assessments to Burundi and the Dominican Republic) Compendium of HIV and HIV/Tubercu- losis Gender Country Assessments, 2013-2015. Forthcoming. UNAIDS. “Three Ones” Key Principles. Geneva: UNAIDS, 2004. UNAIDS. UNAIDS Gender Assessment Tool. Geneva: UNAIDS, 2014. UNDP. On Course: Mainstreaming Gender into National HIV Strategies and Plans – A Roadmap. 2014. Wood, M., E. Ottolenghi, C. Marin, E. Sonneveldt, and J. Yuan. 2007. What works: a policy and program guide to the evidence on postabortion care. Washington DC: United States Agency for International Development [USAID], Postabortion Care Working Group. The Evidence Project Population Council 4301 Connecticut Avenue, NW, Suite 280 Washington, DC 20008 USA tel +1 202 237 9400 evidenceproject.popcouncil.org Acknowledgments List of Acronyms Abstract Introduction Methodology Results Discussion Limitations Conclusion References

View the publication