Table 1.

How Results-Based Accountability Applies to Two Different Types of Government Programs

Case Unit of Accountability/StrategyCity Jobs ProgramResearch Funding Agency
Population (for communities, states, nation)City residents and external businesses that may be attracted to relocateU.S. and global population
ResultsFull employment, economic well-beingNation free of cardiovascular disease
IndicatorsUnemployment rate, annual personal income, job satisfaction ratingsCardiovascular mortality rate
Myocardial infarction incidence and case fatality
StrategiesIdentify causes of unemployment (manufacturing shifts, tax policies, inadequate education)Identify causes (biology of cardiovascular disease, population behaviors, economic and social policies)
Identify new opportunities for employmentIdentify scientific opportunities for investigation
Identify partners (employers, state government, labor unions, schools and universities)Identify partners (researchers, policymakers, physicians, schools and universities)
Establish programs (eg, job-training program)Establish programs (eg, government funding of researchers)
Performance (for programs/agencies)
Identify customersUnemployed workersApplicants/potential investigators
Potential employersUniversities, along with their applicants and grantees
Identify servicesProvide training of new skillsEnable applicants to navigate system by establishing fair competition, answering questions, providing advice
Enable grantees to perform research by giving funds, clarifying rules, monitoring progress, allowing flexibility as appropriate
Identify metrics (over time—Figure 1)
How much?Courses offered (eg, computer programming, basics of laboratory technique)Applications received and reviewed
Number of students enrolledGrants funded
Research funds disbursed
How well?Faculty hired and retainedGrant reviews processed and completed according to schedule
Students who completed courses and passed examsFunds disbursed according to schedule
Progress reports received, reviewed, and acted upon according to schedule
What impact?Number or proportion of students who secured jobsApplicant and grant queries responded to in timely and effective manner
Average income of students who secured jobsClinical trials that recruit subjects and complete processes on time and within planned budgets
Publications (including number, cost per publication, appearance in high-impact journals)
Citations (including number, cost per citation, appearance in major systematic reviews or guidelines, appearance in high-impact journals, h-index)
Commercial (including patents, products readied for clinical testing, cost per patent)
Economic (jobs created, additional business activity stimulated)
Practice or policy (including changes to guidelines or law, changes in evidence-base prescription practices)
Tell the stories behind the metricsAre there measurement artifacts that convey misleading impressions?Are there measurement artifacts that convey misleading impressions?
Were the appropriate skills taught?Is peer review identifying the best scientists and best projects?
Were students correctly matched to courses?Is competition from industry, foundations, and agencies from other countries crowding us out?
Were the right faculty hired?Are budgets appropriate and are they adequately flexible? Do they account for dynamic changes in technologies and prices?
Did we adequately research employer needs?Are powerful economic forces keeping researchers and/or patients out of government-sponsored projects? Are economic forces preventing adoption and dissemination of evidence-based technologies?
Identify partnersLocal employers, educators, experts in program evaluationAcademic leaders, professional societies, journal editors, other government agencies, industry researchers, research leaders, experts in program evaluation
Strategies and action planModify and improve metricsModify and improve metrics
Change hiring criteria for facultyModify peer review criteria
Adopt technologies to improve teaching efficiencyNew program announcements (sometimes with dedicated review) or initiatives
Systematize feedback with employers to enable continuous change and improvementsWorkshops and working groups
Joint programs with other agencies
Establish special advantages (eg, to early stage investigators)
  • Emphasis is on performance accountability. The table provides examples but is not meant to be comprehensive.