Lessons Un-Learned from the Bush and Obama Administrations’ Experiences with Evaluation and Performance Measurement

By Dr. Kathryn Newcomer (Director of the Trachtenberg School) and Nick Hart (PhD '16)
December 11, 2015
The following also appeared in the Italian Evaluation Association Newsletter 

Calls for "evidence‐based policy," "data-driven decision-making,” and for assessing how well government programs work have been around for many years.  The George W. Bush and Barack Obama Administrations both espoused support for the generation and use of evidence to guide and improve government management.  The two presidents brought very different professional experiences, political views, and policy advisors to the job as Chief Executive of the federal bureaucracy, yet their “President’s Management Agendas” established similar expectations about the use of evaluation and performance data.  Attempts to reform Federal performance measurement and evaluation systems under the Bush and Obama Administrations had substantial similarities in fundamental approaches, and both strategies have yielded modest results, for a number of similar reasons. In fact, while it seemed at first that the Obama approach had benefited from lessons learned during the Bush Administration, in fact, many of the same challenges appear to have continued. We identify eight remaining challenges that neither administration fully addressed that relate to the lessons “learned” for each administration.

  1. Calibrate OMB's Role vis-à-vis the Agencies.
    It is necessary to appropriately calibrate the role of OMB vis-à-vis the executive agencies regarding ownership of the performance or evaluation efforts.  OMB has an institutional responsibility to implement the president's agenda, and therefore maintains the power and influence to ensure agency staff are attentive to prescribed requirements.  For example, when agency staff believe that OMB staff will render judgment on the quality or adequacy of analyses—particularly when agencies perceive the OMB judgment criteria as narrow or inconsistent with agency missions and needs—these processes can and often do become merely compliance exercises where agencies minimally participate.
  2. Establish and Sustain an Audience.
    Establishing and then maintaining the attention of an implementing audience for the performance or evaluation initiatives has been challenging for both the Bush and Obama administrations.  In both instances, initiatives became linked closely with certain political appointees within OMB, then when those appointees departed the initiatives became less urgent for agencies.  This may, in part, be due to shifting priorities within OMB, but the effect on agencies is clear.  Further, given competing demands on a wide range of policy and budget initiatives, OMB's small staff has limited capacity to ensure performance measurement and evaluation initiatives receive constant attention and appropriate technical assistance for agencies  charged with translating then implementing OMB's guidance. 
  3. Effectively Implement Initiatives with Multiple Agencies.
    Implementing government-wide, or cross-agency initiatives effectively to improve government performance has been a repeated challenge. Cross-agency collaboration is sorely needed, yet it has long been an oxymoron. To ensure that effective and authentic collaboration happens, substantial funding and staff time are required.  Agencies will need resources and political will to fully collaborate on complex issues that span intergovernmental jurisdictions to be the most effective.
  4. Generate Sufficient Success Stories.
    Neither administration developed and disseminated sufficient success stories to relay to Federal executive leaders, Congress, or the public that the performance or evaluation initiatives “worked.”  Because leadership for performance or evaluation initiatives requires persuasion, sharing of success stories can be beneficial.  But when administration representatives rely on the same handful of stories, which may be generous representations of reality,  agency audiences and even OMB staff can easily become skeptical of the value of the initiatives.
  5. Build Adequate Evaluation Capacity.
    Building adequate evaluation capacity in agencies beyond Education, Labor, and HHS is necessary to support performance and evaluation initiatives over the long-term.  Basic capacity can help agencies identify what data to collect and to appropriately analyze the data to learn important lessons about program implementation and improvement.   Both administrations predominately focused on RCTs for the purposes of impact evaluation.  Such an approach alienated staff in agencies beyond the handful of agencies with sufficient resources and expertise that historically administered RCTs.  Reliance on and calls to increase the use of RCTs have been at the expense of mixed methods for evaluations with formative purposes.
  6. Establish Synergies Between Evaluation and Performance Staff.
    Additional emphasis is needed on creating and institutionalizing synergies among performance measurement and evaluation offices and staff within the agencies.  In virtually all Federal agencies, including Labor and HHS, performance reporting occurs in totally separate offices from evaluation initiatives.  The two activities are viewed by agency staff as separate enterprises, with little interaction, thus performance measurement and reporting do not sufficiently benefit from nor feed into evaluation initiatives.  A separation between performance measurement, on the one hand, and the rest of evaluation practice, on the other, has also been manifest for years in OMB and GAO, both organizationally and in their written products.  Addressing this challenge will be difficult because bureaucratic inertia, cultural norms, and individual worldviews are difficult to modify.
  7. Socialize and Train Political Appointees and Senior Executives Appropriately.
    Neither administration made notable strides in socializing and training political appointees to assume visible and authentic leadership roles in performance or evaluation initiatives.  With typically short tenures, political appointees barely have time to learn the ins-and-outs of the agencies' and programs’ missions and then try to “make a difference.” Performance or evaluation initiatives are typically not among new political appointees' highest priorities.  Targeting training at political appointees through the White House's Office of Presidential Personnel could help address this challenge.  In addition, greater exposure to the initiatives among career senior executives who work and consult directly with senior political officials could also contribute to addressing the challenge.
  8. Consult with Appropriate Congressional Leaders and Staff.
    Neither the Bush or Obama Administration consulted extensively with congressional leaders about the design of performance and evaluation initiatives, and, in turn, congressional staff did not appear to make use of information made publicly available for Congress.  Consulting with interested congressional committees or leaders about the need for or potential value of proposed performance or evaluation initiatives during the design phase would help ensure the initiatives are appropriately calibrated to meet congressional needs.  Presidential management agendas or frameworks, including performance or evaluation initiatives, are just that – they are owned by the President and the executive branch. While GPRA and the GPRA Modernization Act specifically called for more congressional consultation, to date there has not been a strong track record of such collaboration occurring even when there are not ideological differences among the branches.  The breakdown in consultation is perhaps more pronounced when different political parties control the two branches. 

In the era of so-called evidence-based policymaking, the focus on incorporating additional information in to decision-making processes is closely linked to the idea prevalent over past decades that policymakers are able to and capable of identifying such information and the appropriateness for applying such information in a decision-making framework.  At the same time the performance measurement and evaluation efforts of the Bush and Obama administrations have sought to provide some level of accountability, while also describing the current effectiveness and performance of government programs.  But determining what constitutes valid, relevant, and reliable evidence for many Federal programs and agencies is complicated for those staff operating the programs—typically more challenging than anticipated at the outset—and can be even more difficult to interpret for individuals with expansive oversight functions, such as congressional committees and OMB staff.   

Capacity to improve the production of performance measurement and evaluation activities remains limited if not inconsistent across the Federal government.  The U.S. government should continue to explore ways to better develop the capacity to align with the dual purposes defined for pursuing these activities.

 

DISCLAIMER: The views presented in this paper are those of the authors and do not represent the official position of the U.S. Government or the U.S. Office of Management and Budget (OMB).