Total Market Approaches - Directions for the Total Market Approach Stewardship Capacity Tool

Publication date:

Directions for the Total Market Approach Stewardship Capacity Tool Reproductive Health Supplies Coalition Inovation Grant V 1.0 Contract # NOD.1841- 705888-GRT July 2015 Prepared for: Reproductive Health Supplies Coalition c/o PATH - 2201 Westlake Avenue, Suite 200, Seattle, WA 98121 Submitted by: Abt Associates 4550 Montgomery Avenue Suite 800 North Bethesda, MD 20814 1. Overview . 1 1.1 Objective . 1 1.2 Regulation . 1 2. Stewardship . 2 2.1 Definition of Stewardship Capacity. 2 2.1.1 Policy and Dialogue . 2 2.1.2 Regulation . 3 2.1.3 Data Collection and Analysis: . 3 3. The Assessment . 4 3.1 Components of the Assessment . 4 3.2 How it works . 4 3.3 Process . 4 3.3.1 Preparation for the Assessment . 4 3.3.2 Implementing the Assessment . 5 Annex A: Report Template. 6 OVERVIEW Abt Associates Directions for Total Market Approach Stewardship Capacity Tool V 1.0 ▌pg. 1 1. Overview This document contains the directions for identifying and assessing the ability of a single government agency to steward a total market approach for family planning commodities. The operational definition of a TMA is: a strategic and analytical approach to leveraging the comparative advantage of different sectors to meet the needs of the population, in this case, the need for family planning commodities. The “sectors” referred to in this tool are the public sector and the private sector, with the private sector being further distinguished between the commercial sector and nongovernmental organizations (also referred to as social marketing organizations). Stewardship is defined as leading and coordinating a strategy with different sectors. Through a brief preliminary review of family planning activities in the country, an FP planning agency will be identified as the best positioned to steward the TMA. This agency, hereafter referred to as the FP planning agency, will be assessed using the Total Market Assessment Stewardship Capacity Tool (TMASCT). 1.1 Objective The primary objective of the TMASCT is to use indicators to estimate the capacity of a government agency to steward a total market approach for FP commodities. (The TMASCT is an Excel workbook.) 1.2 Regulation The ability of a government to regulate the quality of family planning commodities is instrumental to its ability to steward a TMA. However, regulation is rarely if ever conducted by the agency best positioned to steward a TMA. This tool assesses the FP planning agency’s knowledge of the regulatory function and its ability to access the information it needs to steward a TMA. Informants within the FP planning agency are expected to directly answer questions about the regulatory agency or collaborate with the regulatory agency to do so. If personnel from the FP planning agency are unable to either respond directly or collaborate with the regulatory agency to score the indicators, the evaluator will reach out to an informant in the regulatory agency. STEWARDSHIP Abt Associates Directions for Total Market Approach Stewardship Capacity Tool V 1.0 ▌pg. 2 2. Stewardship 2.1 Definition of Stewardship Capacity The assessment tool was designed to test a government agency’s ability to steward a multi-sectoral strategy to meet the family planning commodity needs of the population, by identifying indicators that succinctly and accurately measure stewardship capacity. Stewardship has been defined as leading and coordinating a strategy with different sectors. For the family planning environment, three responsibilities of stewardship were identified and reflected through basic and appropriate measures of capacity: Policy and Dialogue, Regulation, and Data Collection and Analysis. These categories are intended to capture the full scope of a government agency’s stewardship responsibilities, and the indicators within each category are intended to estimate the government’s ability to meet the responsibilities of each set of tasks. The indicators are further broken into tasks that are necessary to fulfill each responsibility for stewardship. Stewardship Capacity for a TMA for Family Planning Commodities Policy and Dialogue Regulation Data Collection and Analysis Mandate Regulation Data Collection Sufficient Funding Sources Sufficient Funding Sources Data Analysis Recognized Need Legal Framework Data Management Dialogue Data Quality Monitoring and Evaluation Data Dissemination Human Resources Data Use 2.1.1 Policy and Dialogue The Policy and Dialogue category is intended to organize the tasks related to the agency’s ability to convene and guide a multi-sectoral approach. This was selected as a category of responsibility because the government must have the ability and willingness to lead a multi-sectoral team to manage a TMA.  Mandate: Mandate indicators establish the authority of the government to conduct activities that fall under a TMA.  Sufficient Funding Sources: Indicators for reliable funding sources ensure that the government has the resources to fulfill responsibilities related to managing a TMA.  Recognized Need: The Recognized Need Indicator acknowledges that while the infrastructure may exist and make possible the management of a TMA, the agency will successfully manage a TMA only if its leadership willingly undertakes the role of steward.  Dialogue: The Dialogue indicator establishes that the government is capable of leading a multi- sectoral discussion. STEWARDSHIP Abt Associates Directions for Total Market Approach Stewardship Capacity Tool V 1.0 ▌pg. 3  Monitoring and Evaluation: The Monitoring and Evaluation indicator ensures that the project management is tracking progress towards goals and the success of implementation. Human Resources: The Human Resources indicator addresses the need for adequate staffing within the FP planning agency to lead a TMA 2.1.2 Regulation The Regulation category organizes the tasks related to regulation of family planning commodities. As explained above, this tool is intended to measure a government agency’s ability to lead a TMA, thus the Regulation indicators will be addressed to the FP planning agency, not the regulatory agency. This was selected as a category of responsibility because of the importance of regulation in a government’s ability to engage the family planning commodities market.  Regulate: Regulation indicators ensure that the government is fulfilling the responsibility of regulation and that the FP planning agency is aware of these activities.  Sufficient Funding Sources: The regulatory agency’s funding must be sufficient to operate, and sufficiently transparent to be tracked and understood by the FP planning agency.  Legal Framework: There must be a legal basis for establishing and enforcing family planning commodity regulations. These should be understood by the FP planning agency in order to manage a TMA. 2.1.3 Data Collection and Analysis: The FP planning agency must understand the family planning commodities market well enough to understand and meet the needs of the population. Data Collection and Analysis Indicators measure the government’s ability to identify gaps and activate the TMA appropriately and efficiently.  Data Collection: Data Collection indicators identify the data points required, at a minimum, for understanding a family planning commodities market.  Data Analysis: Data Analysis indicators measure the government’s ability to analyze the data collected for use in a TMA.  Data Management: The Data Management indicator addresses the proper management of data from different sources for analysis and reporting.  Data Quality: Data Quality indicators establish the quality and corresponding usefulness of the data that will be collected, reported and used to manage a TMA.  Data Dissemination: Data Dissemination indicators measure the government’s ability to report activities and data to stakeholders.  Data Use: The Data Use indicator measures the government’s ability to use data to make evidence-based decisions to better meet the needs of the population in the area of family planning commodities. THE ASSESSMENT Abt Associates Directions for Total Market Approach Stewardship Capacity Tool V 1.0 ▌pg. 4 3. The Assessment 3.1 Components of the Assessment The assessment is made up of a questionnaire, the tool (i.e., TMASCT, an Excel workbook), and a report template. It uses a mixed (qualitative and quantitative) method to analyze data collected from informants to estimate the government’s ability to steward a TMA. 3.2 How it works Through a brief preliminary assessment, a single agency is identified as the best suited to manage a TMA. The evaluator will use the questionnaire to collect information from informants within the agency to score the indicators. The questionnaire identifies data sources for criteria that are met, and helps collect observations about the obstacles to achieving unmet criteria. Indicators were selected as measurements to ensure that tasks are undertaken, and, consequently, that the responsibilities are managed to enable the government agency to steward a TMA. Each indicator has specific criteria; each criterion met receives one point. These scores quantify results for ease of reference. The qualitative responses to the questionnaire can guide capacity-building interventions. The indicators are conversion indicators (scored as either “yes” or “no”) based on the achievement of criteria. Only if all of the criteria for an indicator are met will the indicator be converted from “no” to “yes.” 3.3 Process 3.3.1 Preparation for the Assessment No. Task 1 With a focal point—i.e., an individual who contributes to the preparation and implementation of the assessment but does not fill in the questionnaire or tool—the evaluator reviews the family planning commodities landscape in the country. 2 The evaluator reviews the government agencies in the landscape and selects the best positioned to steward a TMA; this will likely be a planning agency such as the Maternal and Child Health Directorate or similar government agency. It may also be the Commodity Security Committee or agency. This is the FP planning agency for the purposes of the assessment. 3 The evaluator secures a meeting with the director of the FP planning agency and any necessary counterparts to schedule a time to meet with informants. The assessment components should be shared with the director of the FP planning agency so s/he is aware of what information and data sources are needed, and can identify informants to include in the assessment. The evaluator must make it clear that data sources are crucial to scoring the assessment. 4 The evaluator sets a time and place for the meeting with informants. Depending on the existence and availability of the data sources, the meeting can take several hours. 5 As soon as possible, the evaluator should determine whether the FP planning agency can answer questions and produce data sources for the regulatory functions. If that agency needs to coordinate with the regulatory agency or the evaluator must engage the regulatory agency through different channels, the evaluator should make these arrangements as soon as possible. THE ASSESSMENT Abt Associates Directions for Total Market Approach Stewardship Capacity Tool V 1.0 ▌pg. 5 6 The evaluator should meet with the informants in a room that is quiet and comfortable. 7 The evaluator should bring paper copies of the questionnaire and plenty of notepaper, just in case of hardware failure, but should enter the data directly into the MS Word questionnaire. 3.3.2 Implementing the Assessment Once the background information has been gathered and synthesized, the FP planning agency is identified, and informants are selected and invited to participate. No. Task 1 The evaluator gathers informants from the selected agency in a meeting room or quiet, comfortable place to talk. 2 The evaluator hands out paper copies of the questionnaire so the informants can read and follow along. 3 The evaluator leads the informants through the questionnaire, identifies appropriate data sources for each indicator, and ensures that the sources are evidence for fulfillment of the criteria. Informants should fill in the questionnaire as accurately as possible. It is recommended that the data are filled in directly within the Word document. 4 The evaluator accurately notes why each criterion has not been fulfilled, using the predetermined selections when possible, or filling in a reason if none of these predetermined selections appropriate. 5 The evaluator notes the obstacles to achieving the criterion. In this section the evaluator may also note challenges if the criterion has been met. 6 The evaluator assigns the points to each criterion and enters the point into the appropriate field. 7 The evaluator adds up the criteria points for the total for each indicator to enter into the appropriate field. 8 The evaluator transfers the points to the TMASCT Excel file. 9 The evaluator notes the agency that provided the data source, in the Excel file. 10 The evaluator checks the links and double-checks the calculations. 11 The evaluator fills in the tables in the report template. 12 The evaluator analyzes the results. Evaluators should look at the score for each indicator, and use the qualitative data to justify the results and to suggest short- and long-term areas of improvement. Highlight successes for each responsibility (Policy and Dialogue, Regulation, Data Collection and Analysis). Synthesize the informant’s responses regarding obstacles to explain the failure to meet criteria for each responsibility. Use the specific analysis suggestions in the report template for each section. 13 The evaluator replaces the placeholder text in the report template with analyses. 14 The evaluator identifies short-term and long-term recommendations for improvement. Evaluators should synthesize the informant’s observations about the obstacles with their own, and use the qualitative data collected to explain the weaknesses. ANNEX Abt Associates Directions for Total Market Approach Stewardship Capacity Tool V 1.0 ▌pg. 6 Annex A: Report Template Total Market Approach Stewardship Capacity Tool Background The Total Market Approach Stewardship Capacity Tool was used to assess the stewardship capacity of [placeholder for the FP planning agency name] in [placeholder for the country name] on [placeholder for dates] by [placeholder for name of evaluator]. [Placeholder for country-specific background information: why the government has now chosen to investigate the capacity of an agency to manage a TMA. Use the Directions as an Annex and refer to them here.] Objectives The primary objective of the tool is to use indicators, or measurements, to estimate the capacity of a government agency to steward a total market approach for FP commodities. Findings [Placeholder for observations about the general findings of the tool; lead into the responsibility- specific findings and responses to the following questions:  If a TMA were to be implemented in this country for family planning commodities, could a government agency manage it?  Is the FP planning agency, selected for assessment, potentially capable of running it?  Could the agency steward a TMA tomorrow? If “no,” could the agency steward a TMA with support from another agency? If so, in what way?  What sort of capacity needs to be built in order for this agency to steward a TMA? Justify conclusions with detailed observations of each responsibility. The responsibility is undertaken only when all points possible have been awarded.] Policy and Dialogue Policy and Dialogue Points Possible Points Obtained Percentage Attained? Yes/No Mandate Reliable Funding Sources Recognized Need Dialogue Monitoring and Evaluation Human Resources ANNEX Abt Associates Directions for Total Market Approach Stewardship Capacity Tool V 1.0 ▌pg. 7 [Placeholder for observations on the findings from the Policy and Dialogue component and the government’s ability to steward a TMA; address each indicator in turn that was not attained, and what criteria were not met. What are the strengths or indicators that were earned points?  What are the weak areas—criteria that were not attained?  Why weren’t they attained?  Use the qualitative data to describe the obstacles and possible solutions.  Use your own observations to augment the discussion of obstacles and possible solutions.  What changes need to be made, or what capacities need to be built up, in order to convert the indicator from “no” to “yes,” based on the observations made during the assessment? Specifically, observe indicator 1.4.4 and comment on the ability of the FP planning agency to report on the regulatory function, and its ability to work with the regulatory agency.] Regulation Regulation Points Possible Points Obtained Percentage Attained? Yes/No Regulation Reliable Funding Sources Legal Framework [Placeholder for observations on the findings from the Regulation component and the government’s ability to steward a TMA; address each indicator in turn that was not attained, and what criteria were not met. What are the strengths or indicators that were earned points?  What are the weak areas—criteria that were not attained?  Why weren’t they obtained?  Use the qualitative data to describe the obstacles and possible solutions.  Use your own observations to augment the discussion of obstacles and possible solutions.  What changes need to be made or what capacities need to be built up in order to convert the indicator from “no” to “yes,” based on the observations made during the assessment?] For each indicator note the source: was the data source available directly from the FP planning agency, or was it requested by them from the regulatory agency? If you received the data source from or via the FP planning agency, note the FP planning agency. If you received the data source directly from the regulatory agency without support from the FP planning agency, then note the regulatory agency as the source. ANNEX Abt Associates Directions for Total Market Approach Stewardship Capacity Tool V 1.0 ▌pg. 8 Code Indicator Agency 2.1.1 An agency within the government regulates the quality of family planning commodities distributed in the country. 2.1.2 Laws exist to regulate the quality of FP commodities. 2.1.3 There are procedures to license distributors and retail outlets. 2.1.4 There are procedures to register FP commodities. 2.1.5 The regulatory agency regularly conducts inspections. 2.2.1 The commodity regulatory agency receives reliable funding to operate. 2.2.2 The commodity regulatory agency receives sufficient funding to operate. 2.3.1 Sanctions against noncompliance for FP commodity-related regulations are enforced. Please specifically note each indicator that the regulatory agency needed to address directly. What are your observations? Is the regulatory agency able and willing to work through the FP planning agency? What obstacles exist to collaboration?] Data Collection and Analysis Data Collection and Analysis Points Possible Points Obtained Percentage Attained? Yes/No Data Collection Data Analysis Data Management Data Quality Monitoring and Evaluation Data Dissemination Data Use [Placeholder for observations on the findings from the Data Collection and Analysis component and the government’s ability to steward a TMA. Address each indicator. ANNEX Abt Associates Directions for Total Market Approach Stewardship Capacity Tool V 1.0 ▌pg. 9  What are the strengths or indicators that were earned points?  What are the weak areas—criteria that were not attained?  Why weren’t they attained?  Use the qualitative data to describe the obstacles and possible solutions.  Use your own observations to augment the discussion of obstacles and possible solutions.  What changes need to be made, or what capacities need to be built up, in order to convert the indicator from “no” to “yes,” based on the observations made during the assessment?] Opportunities and Next Steps [Placeholder for general recommendations based on findings—specifically, address each responsibility in turn that was not attained and what the overall issues were. Synthesize the informant’s responses in addition to your own, with respect to obstacles to achieving the criteria.  What are short-term solutions? What can the agency do in the immediate future with no policy or funding changes to improve stewardship capacity?  What are the long-term solutions? What investments of resources are required to meet criteria? What systematic or organizational changes need to be made or what capacities need to be built up in order to improve the stewardship capacity score?]

View the publication