Policy Implementation Assessment: Policy and Program Guidance

More Info

In 2012, USAID published a policy document titled Building Resilience to Recurrent Crisis: Policy and Program Guidance (Resilience Policy) in response to two devastating humanitarian crises — catastrophic drought in the Horn of Africa (HoA) and compounding impacts of recurring drought and conflict plaguing the Sahel. The Resilience Policy defines this concept in the context of USAID and aims to build resilience by increasing the capacity of systems and people to reduce risk and better withstand shocks and stressors. The document articulates the Agency’s vision of using these aims to achieve a sustainable reduction in overall humanitarian need and recurrent crises, which often derail development efforts and outcomes. The policy includes a set of principles to apply across resilience programming. It also includes an operational agenda directed toward increasing effective, multi-sectoral collaboration across humanitarian assistance (HA) and development assistance (DA) actors through: 1) joint problem setting, 2) coordinated strategic planning, 3) mutually informed project design, and 4) robust learning. The agenda applies in a set of Resilience Focus Countries (RFCs), which receive ‘resilience funding’ through a variety of mechanisms (most notably Title II food assistance and agriculture DA), although there is no specific Congressional earmark for resilience.

Purpose & Methodology

This Policy Implementation Assessment (PIA) aims to examine to what extent the Agency has achieved the objectives laid out in this policy through shifts in processes and programs, collect evidence about measurement of outcomes, provide input into the revision of the 2012 Policy and strengthen its future implementation. The mixed-methods assessment was conducted between December 2020 and November 2021 and focused on six assessment questions examining: policy implementation in RFCs; the evidence base for progress toward policy goals; mainstreaming of resilience across the Agency; supportive institutional structures and recommendations for the Policy’s revision. The data collection included: 1) document review of approximately 400 documents, including in-depth scoring of Program Cycle (PC) documents and an overview of technical documents such as evaluations, trainings, and Agency resilience resources, 2) a survey administered to targeted listservs, eliciting 219 responses across USAID OUs, 3) 35 Key Informant Interviews with USAID staff members from Washington Bureaus, field Missions, and the Resilience Technical Working Group (RTWG), 4) mixed-methods deep dives to explore resilience-related work in a purposive sample of OUs and strategic planning processes, 5) analysis of Agency funding data by program area from 2012 to 2020, and 6) a recommendation co-creation workshop conducted with 15 USAID stakeholders to refine, ground-truth, and operationalize draft recommendations.

Key Findings

Application of Operational Agenda Through Joint Planning Cells

In general, USAID’s operational agenda was robustly applied by two regional structures, called Joint Planning Cells (JPCs). JPCs consist of a group of USAID humanitarian and development experts from different disciplines who work together to analyze the root causes of vulnerability in a particular geographic area and develop a resilience strategy and related programming. A JPC was used to develop a regional strategy in the HoA, in response to the devastating 2011 drought, and the process was replicated in the Sahel in early 2012. The JPCs emphasized cross-team collaboration among HA and DA teams and had similar goals to a Regional Development Cooperation Strategy (RDCS) team but operated outside of the PC and focused exclusively on resilience.

The HoA JPC was an effective structure for implementing the Resilience Policy’s operational agenda and its legacy continues through the Horn of Resilience Network (HoRN). It operationalized joint analysis and planning processes across HA and DA teams, designed mutually- informed projects, and adapted programming using robust learning, including evidence forums and workshops. Since its creation, it has expanded to include geographic zones in Uganda, Somalia, and DRC under the follow-on to the JPC, the HoRN, and has also been integrated into Program Cycle processes. The HoRN and resilience-focused activities in the region are currently fully aligned with the Resilience Policy, ensuring that HA and DA programs that make up the resilience portfolio in each country can flexibly respond to the needs of communities and systems during crises. The sequencing, layering, and integrating (SLI) of projects and activities across HA and DA portfolios is a direct result of joint analysis and planning that uses collective impact to promote synergistic effects on resilience in communities. Despite some challenges in data collection, telling the resilience story, and achieving full SLI as planned, the framework for the resilience approach is solidly in place.

The Sahel JPC, established in 2012, was also an effective structure, bolstered by the creation of the Sahel Regional Office (SRO) and drawing on lessons in strategic planning and mutually-informed project design from the HoA. The SRO was tasked with Sahel resilience implementation, coordination between HA and DA actors, and a multisectoral approach with participation of staff from different offices and activities in the field. The intention of this JPC was to develop a comprehensive approach to address the relief-recovery-development nexus(es), deliberately aligning HA with DA at household, community, and systems levels. Geographic choices thus started with the ecological and demographic review – reinforced by existing programming to enhance cooperation and coordination among the activities. The enabling environment included governance at multiple levels as well as regional organizations that already worked on humanitarian and development initiatives. The Sahel JPC also planned for and designed projects that robustly accounted for both HA and DA through the Resilience in the Sahel Enhanced (RISE) I initiative and its follow-on RISE II. Despite joint strategic planning and SLI-centered design, procurement impediments led to some challenges in RISE I, and building a more open network of resilience implementers in each country required time and resources. However, RISE I implementation was well-documented, and lessons learned were integrated into the design of RISE II.

Application of Resilience Policy in RFCS Through the Program Cycle and in Humanitarian Assistance

USAID has applied aspects of the Operational Agenda throughout the PC and HA in RFCs, but experience varies across countries, between new and old RFCs, and among the four components of the operational agenda. Document analysis showed stronger policy alignment at the strategic planning level and less alignment moving down the PC into project and activity design. Even in RFCs, where the leadership support is strong and ties to the Resilience Policy’s operational agenda are explicit, there are challenges to implementation. These include uneven collaboration and disjointed timelines for procurements between DA and HA, which in turn affects joint implementation of resilience programming through projects and activities.

The first two components of the Operational Agenda, joint analysis and strategic planning, were relatively well reflected in the scoring of RFC CDCSs: averaging 2.5 out of 4 and 3.1 out of 4 respectively (with 4 mapping to full policy alignment). Strengthening the resilience of populations or systems in some capacity was the primary focus of IRs in four out of 11 RFC CDCSs and a DO with supportive IRs in five of 11. Most RFC CDCSs have some alignment with the Resilience Policy and use its concepts and approaches, with newer RFCs scoring slightly higher than the original set.

The third component of the Operational Agenda, mutually informed project design, was more challenging, while the fourth component, robust learning, is being included, emphasized, and planned for in RFCs as an overarching approach. The resilience-focused PADs selected for analysis, representing half the RFCs, scored a 2 out of 4 on average for policy alignment, indicating coordination with HA stakeholders and a resilience focus within the Results Framework (RF), but not full coordination or evidence of SLI. The PADs were at times mutually designed by HA and DA teams, but implementation of the various activities was more challenging to SLI in practice. Robust learning for RFCs is standardized through performance indicators and includes a subset of indicators that add on to and align with those already established for FtF.

Evidence Base for Resilience

USAID has been a thought leader in building an evidence base and generating guidance on resilience measurement, although much of this is still being executed and data collection on high-level impact is ongoing. Evidence building has occurred primarily through baseline, midline, endline and recurrent monitoring surveys for impact evaluations, and population-based surveys (PBSs) for FtF and BHA. USAID has also utilized secondary, host-country, and other donor data, performance evaluations of activities, and special studies and qualitative reports. Many of these findings are housed on the ResilienceLinks knowledge management platform and/or the Development Experience Clearinghouse and presented through various fora. The Resilience Evidence forums in the HoA, Washington, and South/East Asia have been particularly successful in generating collaboration, presenting results, and discussing best practices and lessons learned.

The team did not discover sufficient evidence to assess whether USAID’s resilience work has reduced chronic vulnerability across RFCs but did find evidence from individual activities and RFC programs. Capturing the impact of resilience funding remains difficult. It is challenging to develop an indicator set and data collection methodology sufficiently complex to capture the richness of resilience programming, but sufficiently practical to capture and analyze data in a timely manner and harmonize or compare across a wide range of countries. Additionally, the changing environments of the RFCs (sudden onset environmental shocks, expansion of conflict etc.) contribute to data collection challenges in areas where resilience programming is often located.

The Center for Resilience (C4R) and its monitoring, evaluation, and learning contracts have worked to define and refine resilience measurement. In collaboration with L-FFP, the C4R has developed several key metrics for resilience, together with guidance on how to use these tools. The basics of resilience metrics derive from the food security sector, which has developed an array of well-defined indicator and MEL requirements. Using these practices as the foundation, as well as work by other international organizations, these metrics have been refined and expanded since the policy’s publication.

The indicator “humanitarian assistance averted” shows promise but is not yet operational.

Although the data can be collected at the population level through the support of local governments, it cannot be collected at the micro level for an individual or community. USAID has been working with an implementing partner to develop a quantitative methodology to measure the extent to which investments in resilience programming funded by the U.S. Government, over time, effectively reduce the need for HA. USAID plans to make the method available to missions and partners once it is finalized.

Mainstreaming Resilience

Resilience programming was initially targeted at increasing food security to withstand drought, with an emphasis on livelihoods and crisis response, but has expanded much further at USAID, manifesting in OUs focusing on global health; education; conflict prevention and stabilization; democracy, rights and governance; and even in non-RFC Missions. Resilience programming first evolved and took on more comprehensive approaches to food security programs (focusing on utilization of knowledge and access to and availability of food) and then gained traction in other sectors. However, this emphasis on coordination is primarily, if not exclusively, among DA rather than HA teams. Generally, staff agree with the definition of resilience, but practically this translates into different approaches and activities in each sector. The assessment team analyzed approaches in global health, education, market systems and economic growth, examples of areas in which resilience has been mainstreamed at USAID over the past decade. Staff have worked to conceptualize resilience in these contexts, generate guidance and tools, and shape related policies, and are also strongly represented in technical and leadership bodies related to resilience. Some non-RFCs are also intentionally and successfully integrating resilience into strategic planning at the CDCS level, as well as into project and even activity design. Numerous examples of successful integration of resilience concepts and approaches were found in the non-RFC CDCSs, PADs, and solicitations that were manually reviewed, although alignment with the specifics of the policy (e.g., the Operational Agenda) was generally lacking.

Agency Support for Resilience

Implementation of the Policy in RFCs and beyond has benefitted from various agency structures, capacity building efforts and knowledge-management tools. The structures and individuals have included the Resilience Technical Working Group, which works across sectors and OUs, the Agency Resilience Coordinator, the cross-Agency Resilience Leadership Council (RLC), a few select high-level champions, and the C4R as an established and growing entity. The C4R has provided technical assistance for resilience programming to staff in RFCs, as well as others in the Agency engaged in applying resilience concepts. Resilience Learning Events or Workshops and platforms that staff and partners can access for more information (such as ResilienceLinks and FSNetwork) have also contributed to policy implementation.

Leadership was identified as the most critical enabler needed to ensure understanding and integration of resilience both at USAID/Washington and at the Mission level. In particular, the Mission and Deputy Mission Directors are key to supporting Resilience Coordinators (working-level staff designated in RFCs to facilitate policy implementation), who cannot build integration alone. High-level Mission management support is critical for any type of cross-cutting or multi-sectoral initiative because leadership can engage and enforce coordination among the specific technical sectors. This is particularly true for initiatives without dedicated funding streams. Resilience Coordinators can serve as the first line of information and technical assistance, but they can also be seen in the Mission as messengers of Washington-based initiatives that blur results from specific technical sectors, add a layer of coordination, and a ‘flavor of the month’ approach to ongoing development activities. As such, they can be a particular stress point for resilience programming.

An overall disconnect remains between USAID HA and DA operational processes and there are higher-level implications for finding ways to integrate HA and DA more completely. The personnel, resources, and timelines differ between DA and HA programs, even though in some places, operations may overlap geographically and involve the same beneficiaries. Flexibility mechanisms such as crisis modifiers (allowing for shifts between HA and DA) and Refine and Implement approaches (enabling year-long, post-award, co-creation processes and contingency planning) can provide a bridge between HA and DA. However, the mechanisms are not always well understood or utilized, and sometimes even the most well-laid plans fracture in implementation. A guidance document co-developed by the Office of Acquisitions and Assistance, C4R, and PPL has been a valuable resource for Missions looking to embed more flexibility in their programming awards.

Key Enablers and Constraints

The operational agenda articulated in the Resilience Policy has been implemented through JPCs and RFCs due to a set of circumstances and enablers unique to these situations. For RFCs, the most prominent enablers are supportive Mission and office leadership; funding allocated specifically to resilience programming; staff, such as Resilience Coordinators, dedicated to collaboration across technical teams (including humanitarian); support from the C4R; and the ability of staff to understand and apply what USAID means by resilience in planning, design, and implementation of programs. Additional enablers include the RTWG and RLC, concrete resources through learning events, and guidance on how to implement resilience programming in RFCs.

The disconnect between DA/HA processes, culture, and timelines are key impediments for coordination between DA and HA actors and ultimately for Resilience Policy implementation. The main internal constraints are the different mandates of HA and DA, as well as procurement and implementation timelines, country-specific Mission objectives, capacities of partners, limited local systems strengthening, and funding streams. An often overlooked facilitating or impeding factor is the agency and ability of FSNs to take ownership and have decision-making power to implement the policy within a Mission, given their more sustainable and long-standing positions between FSO rotations. Given the long-term nature of achieving resilience, limited procurement timelines are a common impediment to DA, especially in a portfolio that crosses so many sectors and actors. Finally, establishing an evidence base for these efforts and their collective impact is a challenge and it is still being built, although many data collection and methodology development exercises will soon be completed or have been recently.

Recommendations

1. USAID leadership in DC and Missions should elevate their commitment to embedding a resilience approach into top-level Agency and Mission priorities and encourage the allocation of additional funding, or more strategic use of existing funding, for its implementation beyond Focus Countries.
   1. Review existing portfolios in non-RFCs and identify resilience projects and activities.
   2. Develop a “second tier” focus country fund for priority non-RFCs with a clear need for resilience programming.
   3. Assign the RLC a more active role to support resilience policy implementation.
   4. Commit to embedding resilience across the CDCS and portfolio.
   5. Utilize collaboration between C4R, the RLC, and RTWG to develop a Senior
   6. Management Team-Level Resilience Leadership Bootcamp for RFC and DC Leadership (also encouraged for non-RFCs).
   7. Include resilience-specific guidance in USAID planning and reporting processes.
2. USAID leadership at the Agency and OU levels should reduce internal structural barriers, increase incentives, and increase support to both USAID staff and IPs to conduct joint design, planning, implementation, and monitoring and evaluation across technical silos within DA to support resilience programming.
   1. Invest in multisectoral programming in specific geographic zones in an organized, sequential, and complementary way to build individual and systemic resilience capacities.
   2. Incorporate incentives, motives, and mandates for multisectoral planning and co-design into processes like performance reviews through HR, and through COs to incorporate the same into award language.
   3. Integrate a resilience lens and strengthen coordination across key related sectors: agriculture, nutrition, education, health, livelihood diversification, empowerment of women and youth; and strengthen local safety nets and shock-responsive mechanisms.
   4. Focus MEL systems on shared outcome tracking.
   5. Increase collaboration to develop indicators and align reporting processes by DO teams.
3. USAID should pursue ways to strengthen and operationalize coordination between HA and DA, ideally across the HDP nexus, and articulate how to practically facilitate this coordination in the new policy.
   1. Elaborate on the approach for DA/HA coordination for RFCs and Mission-level planning.

   2. Support governments to develop, implement, and mainstream national Disaster Risk Management (DRM) and response policies and systems across sectors.

   3. Develop benchmarks from national level sources to easily identify shifts between HA and DA, monitored by the Resilience Coordinator in RFCs.

   4. Design and elaborate on the operationalization of BHA’s Recovery, Risk Reduction and Resilience Framework in relation to resilience programming and synergies with DA.

   5. Reinforce training and information provided to IPs, AORs/CORs, COs/AOs about shock responsive and adaptive management acquisition and assistance tools.

   6. Assess local systems and structures that can be strengthened to support HA and DA.

   7. Stress adaptive management, flexibility, and preparedness with IPs (e.g. DRM/disaster response).

   8. Deploy relief teams from organizations with DA bases of operations in the region to facilitate quicker set-up and resource transition during times of crisis when feasible and beneficial.

4. In the spirit of elevating Resilience as an Agency-wide priority, the Resilience Coordinators and cross-office structures in all RFCs should be further empowered and supported by Mission and USAID/Washington resources and leadership. Additionally, the Agency should consider appointing multi-sectoral integration Points of Contact in non-RFC Missions, who will focus on cross-sectoral connections across Agency programs using various lenses.

   1. Increase support for the Resilience coordinator (for RFCs) or Multi-sectoral Integration POC (for non-RFCs) to bridge the gap across technical sectors for resilience integration.

5. USAID’s Resilience Policy should be revised to include most sectors and countries in its conceptual framework, while maintaining a set of RFCs (which could be expanded).

   1. Ensure clarity in policy operationalization for the food security sector in emergency and development settings

   2. Ensure the policy is general enough for application across the Agency through a conceptual framework to illustrate how other sectors fit into resilience.

   3. Address climate change, disease outbreaks and global health security, and violent conflict and peacebuilding in such a conceptual framework.

   4. Develop accompanying guidance with sector-specific examples, emphasizing strengthening resilience and local systems across sectors.

6. The C4R, RTWG, and associated Bureaus should make resilience more accessible and practical through guidance accompanying the revised Policy, which interprets existing sectoral concepts and approaches through a resilience lens.

   1. Develop more guidance with examples and good practices for coordinating implementation of resilience broadly at the Agency level and for individual sectors (without being prescriptive) across HA and DA, and between different types of DA.

   2. Identify opportunities for sequencing, layering, and integrating) DA and HA activities, considering the entire resilience-adjacent portfolio and contract mechanisms for resource shifts.

   3. Develop an online Resilience Toolkit with guidance for multisectoral coordination and sectoral sub-pages that include links to specific guidance, policies, examples, and platforms.

7. The C4R and RTWG should support RFC Missions to develop an overarching Resilience RF for their portfolios and develop accompanying how-to guidance.

   1. Capture the interrelationships between sectors and sectoral outcomes that come together to achieve overarching resilience objectives in these RFs.

   2. Develop a Resilience RF and indicator handbook with sector-specific examples of activities that contribute to resilience for non-RFC and food security actors. This can be part of the Toolkit.

8. USAID should develop or articulate clear definitions for how resilience is measured, including through both sector-specific and more general indicators, tied to the conceptual framework. This will allow portfolios to capture this nuanced concept at the Strategy, Project and Activity levels of the PC.

   1. Use clear definitions and guidance at each level of the PC.

   2. Develop multi-sectoral measurement that encourages adaptive management at the CDCS level, prioritizing the use of complexity-aware monitoring tools and data collection.

   3. Promote collaboration within Missions for multi-sectoral measurement approaches.

   4. Refresh and amplify existing measurement tools with guidance for addressing significant changes in population within the resilience zone.

This publication was produced by Dexis Consulting Group for the United States Agency for International Development. The Dexis team included Stephanie Monschein, Team Lead; Alice Willard, Resilience Subject Matter Expert; Megan Powell, Research Assistant; Porter Bourie and Chris Ying, Technical Support; and Douglas Baker, Project Manager. The authors’ views expressed in this document do not necessarily reflect the views of the Agency or the United States Government.