In the 1990s, Abe Wandersman, PhD, spent a lot of time evaluating the work of community coalitions that had come together to combat substance abuse. What he found was disheartening: The initiatives didn't get the outcomes they wanted.
"After several years and sizeable amounts of funding, the results were very modest," says Wandersman, a psychology professor at the University of South Carolina.
That experience got him wondering why such efforts—and so many like them—were disappointing. That was the catalyst for Getting To Outcomes® (GTO), an accountability program he developed with former students Matthew Chinman, PhD, now a behavioral scientist at the RAND Corp. and VA, and Pamela Imm, PhD, an independent evaluation consultant. The technique, registered by the University of South Carolina and RAND, gives communities, nonprofit groups, government agencies and others a systematic way to ensure they achieve the results they want.
Today's bad economy has made GTO especially timely, says Wandersman. Lean budgets have only intensified the push for accountability that has been growing for years.
"People want to know their money is being well spent," he explains, citing efforts to prevent AIDS, teach people to read or have fewer kids thrown out of school for conduct disorders. "What hasn't been clear is how you get there."
GTO guides users through the entire process, from conducting an initial needs assessment to ensuring sustainability. So far, the approach appears to be improving outcomes for a variety of initiatives, including substance abuse programs and a teen-pregnancy prevention campaign. The American Evaluation Association recognized the approach's impact and honored the GTO manual—Getting to Outcomes 2004: Promoting Accountability Through Methods and Tools for Planning, Implementing, and Evaluation (RAND Corp.)—with its Outstanding Publication Award in 2008.
"That was the icing on the cake," says Wandersman. "What we really get reinforcement from is that people in the field are really putting this work into practice."
Bridging science and practice
To Wandersman, GTO is the natural outgrowth of his career-long interest in bridging the gap between science and practice.
Too often, he says, both researchers and the people tackling a community problem focus on the science part of the process to the exclusion of everything else.
"Researchers say, 'We did the research, we know what works, just do it,'" explains Wandersman. "It's as if the researchers have wrapped these really valuable things in Christmas paper and given them to practitioners, and all the practitioners are supposed to do is unwrap them."
In reality, he says, there's a lot more to launching a program than that. While drawing on the evidence base is necessary, he says, it's not enough. In fact, that step is just one in the list of 10 steps outlined in the free GTO manual, downloadable at www.rand.org/pubs/technical_reports/TR101.
That manual encourages users to answer what Wandersman calls "taxpayer questions"—questions that help ensure programs do what they're supposed to do:
• What are your community's underlying needs, conditions and resources? This question gets users to assess their needs and resources.
• What are your goals, target populations and objectives? Before they go any further, users must have clear answers to these questions.
• Which evidence-based practices can help you reach your goal? This is the step where users survey the scientific literature.
• What actions do you need to take to ensure the selected program fits the community context? Users must consider whether a proposed program will add to existing efforts, duplicate them or even undermine them. At one GTO training session, for instance, two participants confessed that their schools had both abstinence-only sex ed programs and free condom distribution programs. "You have to make sure programs aren't working against each other," says Wandersman.
• What organizational capacities do you need to implement the program? "Evidence-based programs are created with multiple millions of dollars by researchers in carefully controlled conditions, and then we tell communities with a fraction of the budget and without the necessary skills to replicate them with fidelity," says Wandersman. It's not enough to choose a "gold standard" program, he emphasizes. You have to have the capacity to actually pull it off.
• What is your plan for this program? At this point, users know what they're planning to do. This is where they get into the nitty-gritty of how they're going to achieve their goals—the who, what, when and where.
• How will you assess the program's quality and implementation? "Sometimes people work a long time on a strategic plan, but the implementation is lackadaisical," says Wandersman.
• How well did the program work? This is where users evaluate whether or not they made a difference.
• How will you incorporate continuous quality improvement? In this step, users explore whether there are better ways to do things the next time around.
• If the program is successful, how will you sustain it? Users should have a plan for what happens after the initial funding runs out. "Is there anything left?" says Wandersman. "Or is it as if nothing was ever there; it just disappears because the money disappeared."
Evidence suggests GTO is working. In a study published in 2008 in the American Journal of Community Psychology (Vol. 41, No. 3-4), for example, Chinman, Wandersman, Imm and colleagues at RAND compared substance abuse prevention programs that used GTO with those that didn't. Those that used GTO improved their capacity to do needs assessments, set goals and perform the other skills in GTO, says Wandersman. Their program quality improved. And they demonstrated better outcomes.
"We don't know if they actually got greater outcomes because of doing the GTO work or they were better able to document that they had outcomes," says Wandersman. "Both are important to do, because it's much more helpful if you can show people that you're really making a difference."
Teen pregnancy prevention is another success story, says Wandersman.
Wandersman is working with the Centers for Disease Control and Prevention on a teen pregnancy prevention project that uses GTO to promote science-based approaches. In one rural South Carolina school district, for instance, a teen pregnancy prevention organization that used GTO saw pregnancy rates drop from 19 in 2006 to two last year. "One of the things I most like about GTO is that it gives organizations that have limited experience with evaluating programs a clear roadmap and some tools for evaluating their program's success," says psychologist Christine Galavotti, PhD, chief of the Applied Sciences Branch in the CDC's Division of Reproductive Health. "It shows them that by systematically following a few key steps, they can not only greatly improve the quality of program implementation, but they can also see whether or not they are making progress toward achieving their program goals."
GTO is also being used in campaigns to prevent underage drinking and to create individualized treatment plans for children and families in mental health clinics and substance abuse centers.
Wandersman and his collaborators have big plans for the future. Matthew Chinman is leading a randomized controlled study exploring GTO's use in promoting positive youth development. Chinman and Imm are leading another randomized controlled study assessing how GTO can support efforts to prevent underage drinking. And Wandersman is working on collaborations to expand GTO's reach into new content areas, such as children's mental health services, medical care, and systems of care at the Substance Abuse and Mental Health Services Administration.
"We are excited to offer GTO to our grantee communities across the country," says Gary M. Blau, PhD, chief of SAMHSA's Child Adolescent and Family Branch.
"You can use GTO for just about anything—including planning a vacation," says Wandersman.
Rebecca A. Clay is a writer in Washington, D.C.
Letters to the Editor
- Send us a letter