Why Should CBOs Evaluate?
Limited funding for social programs means Community Based Organizations (CBOs) are forced to compete for grants. Evaluation is a way to not only be accountable to funders, but also a way to verify program impact in order to obtain funding in the future. Funders, such as public or private foundations, want to ensure that their money is benefitting the community. Program evaluation is a way to prove that your program is making an impact.
An additional benefit of program evaluation is to ensure that vulnerable populations who bear the burden of HIV are receiving the support and services they need. By evaluating your program you are able to easily identify program strengths and weaknesses which will allow for adaptation and improvement of service delivery mechanisms when necessary.
If you would like a comprehensive guide to evaluation we recommend the following resources:
Evaluating CDC-Funded Health Department HIV Prevention Programs (PDF, 3.7MB)
Innovation Network's Evaluation Plan Workbook (PDF, 978KB)
W.K. Kaiser Family Foundation’s “Evaluation Handbook” (PDF, 328KB)
Need assistance in evaluation?
Contact the BSSV program to find out how to receive free capacity building assistance from an expert in the field. Email or call us at (202) 218-3993.
Disclaimer: This site contains HIV prevention messages that may not be appropriate for all audiences. If you are not seeking such information or may be offended by such materials, please exit this website.
Telling Your Story
Being able to tell funders why your program is both unique and effective is a good way to maintain your funding, as well as ask for more. Your organization’s “story” is a narrative telling how your organization is both unique and effective. In your story you might want to include topics such as:
Who you serve?
Eg: Minorities, high risk populations, hard to reach, etc.Where you work?
Eg: Rural areas, migrant camps, street outreach, etc.When you provide services?
Eg: After hours, weekends, 24 hour, etc.Why your organization began?
Eg: No other services, community worked together, a passionate founder, etc.
Below is a fictional example of an HIV CBO’s “Story.” Notice how it brings out unique aspects of the CBO, as well as highlights the work they are doing in the community.
Example: Unique Story for the Women’s Circle Case Study
“At Women’s Circle we are devoted to preventing HIV and supporting those who are positive. It was started 1995 by Linda John, an HIV positive woman, who saw few opportunities for other positive women of color to find support. Women’s Circle began as a weekly coffee club. We have grown a lot since then! Now we have classes on HIV, like SISTA, 5 support groups, HIV testing, and a group of women who go to the local beauty parlors to do outreach. Although Women’s Circle is in a mostly white, upper class city, most of the women who visit are of color. I think this shows how hard we work to offer things the community wants. At the office we joke that once a woman comes in, she’s with us for life! Someone will come in for testing or a support group, and keep coming back for the other programs. Women say they feel like Women’s Circle is another home. They love the classes and often bring their friends. I love working for a place that really helps women in our community.” |
Each CBO is unique, and each story is unique. Stories can take on many different formats other than the “Women’s Circle” narrative.
To learn more about how to develop your story using logic models and data to influence others, a detailed workbook from the Center for Disease Control and Prevention is available below. This workbook focuses specifically on creating and formatting your program’s unique story. Please note, it has been developed for oral health programs, but the information is equally applicable to HIV programs.
Centers for Disease Control and Prevention’s guide “Impact and Value, Telling Your Program’s Story” (PDF, 585KB)
Need assistance in creating a unique story for your program?
Contact the BSSV program to find out how to receive free capacity building assistance from an expert in evaluation. Email or call us at (202) 218-3993.
SMART Objectives
Setting objectives is a way for your organization to determine what you would like to achieve and in what time period. Often objectives are broad, but SMART objectives provide a structure which ensures the objective is specific enough to know if it has been reached. SMART is an acronym that stands for:
Specific (What is the specific task?)
Measurable (What are the standards or parameters?)
Achievable (Is the task feasible and easy to put into action?)
Realistic (Are sufficient resources available?)
Time Period Specific (What are the start and end dates?)
SMART objectives will not only set the intentions for your organization, they will also appear in your program logic model.
Below is an example of questions you may ask when developing your program’s SMART Objectives along with an example that follow our “Women’s Circle” example.
Example: SMART Objectives for the Women’s Circle Case Study
Key Component
| Questions to Ask Yourself | SMART Objective |
| Specific - What is the specific task? |
| Implementation of the evidence based curriculum, SISTA. |
| Measurable - What are the standards or parameters? |
| A total of 400 women who have completed the curriculum. We hope to have 34 sessions with 12 participants in each session. |
| Achievable - Is the task feasible? |
| There is a large demand for the course from our clients and we had 320 women complete the curriculum last year. |
| Realistic - Are sufficient resources available? |
| Yes, we have been funded for the year and we have 3 facilitators who have already completed training for the program. |
| Time-Bound - What are the start and end dates? |
| One year—between 2011 - 2012 |
| SMART Objective: Between 2011 and 2012, Women’s Circle will graduate 400 women from SISTA. | ||
For guidance on how to formulate SMART objectives, refer to this CDC Resource Kit.
For specific guidance on developing SMART objectives for a DEBI, refer to the SIHLE Monitoring and Evaluation Guide, page 21.
Need assistance in writing SMART objectives?
Contact the BSSV program to find out how to receive free capacity building assistance from an expert in evaluation. Email or call us at (202) 218-3993.
Logic Models
Logic Models illustrate how a program is supposed to work in terms of the logical relationship between resources invested, activities implemented, and changes that result. They help show how your programing activities connect to your SMART objectives and how you will reach your program’s greater goal—such as “Reduce the Incidence of HIV among Women of Color.” Additionally, they are able to effectively summarize your program to a reader/funder more efficiently than a full report or proposal.
Developing a program logic model before planning your evaluation is highly recommended. Logic models allow for all persons involved to see the expected achievements based upon the chain of events that link each element together. Further, they help focus the evaluation plan by identifying what needs to be evaluated, and where to extract the data from along the program implementation process. These elements include:
- Inputs - includes all the human, financial, organizational, and community resources available to direct toward doing the work for the program.
- Activities - includes the processes, tools, events, technology, and actions that are an intentional part of the program implementation.
- Outputs - includes types, levels, and targets of services to be delivered by the program (e.g., what are direct products of program activities).
- Outcomes (SMART Objectives) - includes the specific changes in program participants’ behaviors, knowledge, skills, status, and level of functioning as a result of activities.
- Impact - includes the intended or unintended change, most often a SMART objective.
By developing a logic model, organizations are able to think about the program in ways that prompt the clarity and specificity required for success, and often demanded by funders and potential partners. A logic models allows for:
- An inventory of what organizations have and what they will need to operate the program effectively;
- A strong case for how and why the program will produce desired results;
- A clear sense of the activities necessary to achieve program success;
- A method for program management;
- Clear description of the data necessary for program assessment; and,
- A SMART objective to work towards.
Below is an example of just one piece of the “Women’s Circle” logic model. To review: first the story of “Women’s Circle” was written to highlight what makes the organization unique and effective. In the logic model the inputs, activities, and outputs that support the SMART objective are tied together.
Example: Logic Model for the Women’s Circle Case Study
| Inputs | Activities | Outputs | Outcomes (SMART Objective) | Impact |
|
|
|
|
|
For a detailed explanation of how to develop a logic model, along with examples and checklists, refer to the Logic Model Development Guide written by the W.F. Kellogg Foundation.
If you would like an explanation of logic models in the context of a DEBI, refer to the SIHLE Monitoring and Evaluation Guide, page 8.
- Evaluation Field Guides for DEBIs assigned to the BSSV Program
- Healthy Relationships
- Many Men Many Voices (3MV) (PDF, 71KB)
- SISTA (PDF, 82KB)
- CLEAR (PDF, 95KB)
- Nia (PDF, 156KB)
- Community PROMISE
Resource
Inovation Network's Guide for Logic Models (PDF, 474KB)
Need assistance in creating a logic model for your program?
Contact the BSSV program to find out how to receive free capacity building assistance from an expert in evaluation. Email or call us at (202) 218-3993.
Evaluation Plans
A well-developed evaluation plan will provide meaningful feedback about a program’s efforts. It will focus an organization’s resources and efforts on evaluating a program efficiently based on utility, relevance, and practicality and not the collection of useless and boring data that provide no clear conclusions.
Using a logic model to guide the development of the evaluation plan helps determine specific aspects of the evaluation component. A well-developed logic model can help save time during other phases of program implementation and evaluation. A general rule of thumb is to have at least one evaluation point for each outcome and/or impact listed on the logic model. The logic model can help guide evaluation plan development by providing the following:
- Focus - helps the evaluation team determine what needs to be evaluated.
- Questions - help the evaluation team determine what evaluation questions to ask.
- Indicators - help decide which data to collect in order to answer the questions about program efficacy.
- Timing - helps decide when to collect data during the program implementation.
- Data Collection - helps determine the data sources, methods, instruments, and samples to be used to collect and manage the data.
The chart below provides a portion of the Women’s Circle evaluation plan. The series of charts you complete for each evaluation question is part of the evaluation plan.
Example: Evaluation Plan for the Women’s Circle Case Study
| Evaluation Question: Do participants who enroll in Women’s Circle complete the program? | ||||
| Process Objective | Measure | Data Collection Method/Source | Who will collect the data? | Timeframe for Data Collection |
|
|
| Facilitator | Ongoing |
For specific guidance on developing SMART objectives for a DEBI, refer to the SIHLE Monitoring and Evaluation Guide, page 90.
Resource
Innovation Network's Evaluation Planning Workbook (PDF, 978KB)
Need assistance in evaluation?
Contact the BSSV program to find out how to receive free capacity building assistance from an expert in evaluation. Email or call us at (202) 218-3993.
Role of Data
Data simply means "pieces of information." It can be numbers, words, sounds, or images. Sign in sheets, intake forms, satisfaction surveys- all of these items can provide useful information for an evaluation. Data makes your story stronger and more convincing. Data is collected through program evaluation.
Why is adding data important?
- It gives your audience a better idea of how many people you serve, services you provide, etc.
- It gives credibility to your work
- It makes your organization look good
- It is important to funders and often required (NHM&E requirements for CDC funded CBOs)
Beyond funders’ requirements, there are pieces of data your organization may be specifically interested in. Your program logic model can help determine the specific information you would like gathered to both prove program outcomes and enhance your story. They may overlap with your funder’s requirements. Below we refer back to the “Women’s Circle” story to show the data the CBO feels is important to highlight how they are unique and effective.
Example: Data-infused Unique Story for the Women’s Circle Case Study
“At Women’s Circle we are devoted to preventing HIV and supporting those who are positive. It was started 1995 by Linda John, an HIV positive woman, who saw few opportunities for other positive women of color find support in their struggle. Women’s Circle began as a weekly coffee club. We have grown a lot since then! Now we have:
Although Women’s Circle is in a city that’s 85% white, 95% of our clients are of color. We are really connected to the community! At the office we joke that once a woman comes in, she’s with us for life! Someone will come in for testing, or a support group, and keep coming back for the other programs. 75% of our clients come back at least 3 times. Women say they feel like Women’s Circle is another home. They love the classes and often bring their friends. Among new clients 45% have been referred by a friend. I love working for a place that really helps women in our community”
|
Need assistance in evaluation? Contact the BSSV program to find out how to receive free capacity building assistance from an expert in evaluation, or call us at (202) 218-3993.
Data Collection
By following your program’s evaluation plan you will be able to collect relevant and pertinent data for your program. The sources of data and method of data storage is unique to each organization. A data collection plan will help ensure that your organization is collecting data that will help answer your evaluation plan questions. The data collection plan will include information for each SMART objective that lists where the information will be extracted from, who will be in charge of collecting the information, and the timeframe for information collection. As seen in the unique story infused with data, information can strengthen what makes an organization unique and effective.
Creating your Data Collection Plan
Now that you have determined what pieces of data demonstrate how your organization is unique and effective, formatted those data points into SMART objectives, and aligned them with your program’s activities it is time to implement the program and being to collect the data. The chart below will assist you in determining how and when to collect each piece of data. There should be a chart for each SMART objective. The chart below has been filled in with information from the “Women’s Circle” example. The series of charts you have completed for each SMART objective is a data collection plan.
NHM&E
Funders often require you record specific data points. The CDC is an example. The CDC collects specific pieces of data called the National HIV Monitoring and Evaluation variables (NHM&E) from funded community based organizations. Though these specific data are required by the CDC, some of the required variables may be pieces of data you want to include in your story. For example, those who are implementing SISTA are required to report the total number of clients attending the sessions. You can use the data entered into this variable for your story. NOTE: Not every chart will have a NHM&E variable.
Example: Women’s Circle Case Study Data Collection Plan
| SMART Objective: Between 2011 and 2012 Women’s Circle will graduate 400 women from the SISTA DEBI | |||||
| Process Objective | Measure(s) | Data Collection Method(s)/Source | Who will collect the data? | Timeframe for Data Collection | NHM&E variable |
|
|
| Facilitator(s) | End of each cycle of SISTA | F05 |
If your organization needs assistance in data collection, storage, or analysis submit a CRIS request for assistance.
The chart is from the SIHLE Monitoring and Evaluation Field guide. The Evaluation Implementation Planning Tool, along with more detailed instructions is located on page 85.
More examples of DEBI specific NHM&E variables are located in the SIHLE Monitoring and Evaluation Guide beginning on page 124.
If you are unsure whether or not you are required to collect NHM&E data contact your CDC or health department project officer. To identify the required NHM&E variables for each DEBI, refer to the NHM&E site for the specific DEBI. If you have questions on NHM&E data collection guidance contact the NHM&E Service Center.
Community Data Resources
You may have some data points which are about your community, and not your programs. For example Women’s Circle states “Although Women’s Circle is in a city that’s 85% white, 95% of our clients are of color. ” Below are some sources of community data.
State Level HIV Data, Kaiser Family Foundation
Community Level Demographic Data, US Census Bureau, American Fact Finder
Behavioral Risk Factor Data, including HIV Related Behaviors, Centers for Disease Control and Prevention, Behavioral Risk Factor and Surveillance Survey
Youth Behavioral Risk Factor Data, including HIV Related Behaviors, Centers for Disease Control and Prevention, Youth Behavioral Risk Factor and Surveillance Survey
Maps of National, State, and Community HIV Data, AIDSVu
EpiInfo, free data collection and analysis software, Centers for Disease Control and Prevention
Need assistance in evaluation?
Contact the BSSV program to find out how to receive free capacity building assistance from an expert in evaluation. Email or call us at (202) 218-3993.
Presenting Your Story
Now that you have determined the story you’d like to tell about your organization and have the data to support that story, it is time to write your story in a way to grab others’ attention. You want to tell your story to people or organizations who can:
- Provide more funding
- Donate
- Volunteer
- Change policy
Who Should Hear Your Story?
| Who | Why you should tell them | What they care about |
| Clients | Retention, you care about them | What is going well in the program, data that applies to them |
| Staff | Improve program delivery, motivate staff, reduce turnover | Staff would like to know if they're doing a good job, or where they can improve |
| Funders | Keep your current funding or get additional funding | Program is doing what they requested, money is being saved/spent wisely, population is being reached, story is unique |
| Community | Request donations, new volunteers, gain community support, raise awareness | What the program is doing, who is it helping, how much it is helping them, that the money is being spent wisely |
| Local or National Politician | Get additional funding, increase funding for general HIV services, increase awareness of program | Program is improving their city/district/state, money is being spent wisely, makes them look good |
There are a variety of ways to format your story. You can use the same data in each format. The goal is to present the information in a way that best fits your audience. Examples of formats include:
- One page summary
- Elevator speech (a 30 second speech)
- Newsletter
- Website
- Presentation
- Data summary
For more information on how to select the format of your story, along with some examples, refer to page 5-10 of the Centers for Disease Control and Prevention’s guide “Impact and Value, Telling Your Program’s Story.” (PDF, 585KB)
Need assistance in evaluation?
Contact the BSSV program to find out how to receive free capacity building assistance from an expert in evaluation. Email or call us at (202) 218-3993.
Program Development Resources
- Replicating Effective Programs (REP) Project
- Looks at replication/translation of interventions with demonstrated effectiveness by the CDC. - Diffusion of Evidence Based Interventions (DEBIs)
- Designed to bring science-based, community, group, and individual-level HIV prevention interventions to community-based service providers and state and local health departments by the CDC. - Characteristics of Reputationally Strong Programs (C-RSP) Project
- Identifies common characteristics among “reputationally strong” programs by CDC - CDC compendium of effective HIV prevention programs for 2008
- Contains an overview of evidence-based HIV behavioral interventions. - SAMSHA’s National Registry of Evidence based Programs and Practices (NREPP)
- A searchable online registry of interventions supporting mental health promotion, substance abuse prevention, and mental health and substance abuse treatment. - SAMSHA’s Guide to Evidence-Based Practices (EBP) on the Web
- Contains information about specific evidence-based practices (EBPs) or provide comprehensive reviews of research findings.
