Using Process Evaluation During Organizational Intervention Processes

Presenters: Karina Nielsen, PhD, National Research Centre for the Working Environment, Copenhagen, Denmark; and Ray Randall, PhD, Occupational Section, School of Psychology, University of Leicester, UK

Workshop Length: 3 hours

Workshop Description: Theories that describe the links between work design and employee health indicate that organizational interventions could be effective in improving health and satisfaction at work. There is now considerable research that shows that organizational interventions have inconsistent effects. This may be due in no small part to the influence of factors found in the intervention process. These include organizational changes that are concurrent to the intervention (such as re-structuring), employees’ resistance to change and faulty intervention delivery. Understanding the influence of these factors requires suitable process evaluation methods. Established methods of evaluation focus almost exclusively on effect evaluation (i.e. these deal with the question ‘what changed?’). Therefore, there is a need for process evaluation that also explores questions such as ‘why and how did the intervention work (or fail)?’ This workshop is designed to provide participants with a clear understanding of the nature of intervention process factors and how process evaluation data can be collected. We will use an illustrative case study example to allow participants to explore the challenges of process evaluation. This will also provide workshop participants with information about how such challenges may be met.

Workshop Learning Objectives:

  1. Identify the sources of process evaluation data that need to be collected in order to evaluate fully organizational interventions

  2. Prepare a plan for the collection and analysis of process evaluation data from information given in a case study of teamwork implementation

  3. Describe and critique plans for process evaluation based on a case study of teamwork implementation

From Research to Practice: Creating Age-Friendly Workplaces

Presenters: Janet Barnes-Farrell, PhD, University of Connecticut, James Grosch, PhD, National Institute for Occupational Safety and Health

Workshop Length: 3 hours

Workshop Description: The unprecedented aging of the workforce in the U.S. and many other countries poses both challenges and opportunities for occupational health psychologists.  This workshop focuses on better understanding the often subtle impact of age on workplace outcomes such as job stress and its associated consequences for worker health and well-being.  We will also discuss the Finnish concept of workability as a vehicle for thinking about aging and work.  During the workshop, we will identify workplace barriers that affect many older workers (e.g., changes in technology, age discrimination), as well as practical strategies (e.g., training programs, workplace flexibility, ergonomic redesign) that are currently being used in different industries to promote the health, safety, and well-being of all workers, regardless of age.

Learning Objectives:

  1. Be able to describe current knowledge regarding the association between worker age and workplace outcomes such as productivity, job stress, workplace injuries, work ability, and turnover.

  2. Be able to identify working conditions that may pose a barrier for older workers in terms of their overall job performance and well-being.

  3. Be able to describe the pros and cons of different workplace strategies for creating age-friendly workplaces.

  4. Be able to apply concepts and models discussed in the workshop to actual work environments familiar to the participants.

Integrating the Science of Team Training to Create Workplace Health & Safety

Presenters: Heidi B. King, MS, FACHE, Deputy Director, U.S. Department of Defense Patient Safety Program; and Sallie J. Weaver, MS, University of Central Florida Institute for Simulation & Training

Workshop Length: 3 hours

Workshop Description: This workshop provides an interactive opportunity for participants to learn how teamwork and team training can function as an economically sound method to facilitate a healthy and safe workplace. Heidi King, Deputy Director of the US Department of Defense Patient Safety Program and Sallie Weaver of the University of Central Florida Institute for Simulation & Training will engage participants in a hands-on experience dedicated to developing a plan for optimizing team functioning rooted in nearly 30 years of scientific research.  Participants will review the core competencies of effective teamwork and appraise which are most relevant for creating a healthy work environment in their own organization. They will define the steps in developing and implementing a team training program based on evidence-based practices, as well as identify facilitating factors and barriers in the development, implementation, and sustainment of effective teamwork in their organization. Finally, participants will compose a plan for developing effective teamwork within their organization.  Behavioral tools and implementation stories from TeamSTEPPS, an evidence-based team training program widely used within healthcare, will be used as examples throughout, however the methods and issues addressed in this workshop are applicable to any teamwork competency model as well as a broad range of instructional strategies.

Learning Objectives: Upon completion of the proposed workshop, participants will be able to:

  1. Discuss the relative merits and limitations of teamwork and team training.

  2. Describe the core competencies of effective teamwork and determine which are most relevant for creating a healthy work environment in their own organization.

  3. Define the steps in developing and implementing a team training program based on best practices.

  4. Compose a plan for developing effective teamwork within their organization that identifies potential organizational, team, and individual factors that may help (or hinder) the development, implementation, and sustainment of effective teamwork in their organization.

Integrating Health Protection and Health Promotion: New Approaches to Worker Well-Being

Presenters: Robert Henning, Ph.D., CPE, Associate Professor, Department of Psychology,  University of Connecticut; Michelle M. Robertson, PhD, CPE, Research Scientist,  Liberty Mutual Research Institute for Safety; Nicholas Warren, ScD, Associate Professor of Medicine and  Ergonomics  Coordinator at the Ergonomic  Technology  Center of Connecticut, University of Connecticut Health Center; Suzanne Nobrega, MS, Project Manager of the R2P Toolkit Project, Center for the Promotion of Health in the New England Workplace, University of Massachusetts; L. Casey Chosewood, MD, Manager of the NIOSH Worklife Program, NIOSH; Karen Hopcia, ScD, ANP-BC, Project Director, “Be Well at Work Well” Study, Partners Healthcare System /Harvard School of Public Health; Caitlin Eicher, ScM, Research Assistant, “Be Well Work Well” Study, Dana-Farber Cancer Institute/Harvard School of Public Health

Workshop Length: 3 hours

Workshop Description: This workshop introduces participants to the NIOSH WorkLife program, a research-to-practice effort that provides concepts, policies, and interventions that integrate the principles of traditional occupational health and worker protection programs with proven, innovative health promotion interventions. The presentation will describe essential elements of successful worksite health and wellness programs, powerful policy and built environment levers, real-life examples of integrated worker protection and health promotion programs, and case examples that highlight best practices.  A discussion of special demographic challenges facing employers as they attempt to promote health and build a wellness culture will be included.  Concepts of culture-building, novel strategies for engaging workers, and mechanisms to permanently sustain programs will be elucidated.  Toolkits, web resources, cost and ROI calculators will be shared.

We will also review the newly released worksite health intervention reviews from CDC’s Guide to Community Preventive Services, examining the growing evidence base for work-based prevention interventions shown to be both successful at improving health and leading to a positive return on investment. Case examples and research to practice efforts from NIOSH’s WorkLife Centers of Excellence will also be highlighted, including an overview of integrated worker protection and health promotion programming research now underway and a summary of liaison efforts with community employers, state and local health departments, healthcare institutions, and other partners in this effort.

Workshop Learning Objectives:

  1. Describe the elements of integrated worksite health and wellness that address both work based and non work-related health risks impacting the wellbeing and productivity of the workforce as well as cost to employers.

  2. Identify worker protection, health promotion and prevention resources from NIOSH, CDC and other credible sources

  3. Use research to practice tools including the “Business Decision Scorecard” for intervention planning and business decision making
    Understand the value of engaging employees in the identification and prioritization of health/safety issues/concerns, and in the intervention planning process.

  4. Apply employee data to the intervention development process.

  5. Describe approaches to health promotion and prevention activities such as a participatory approach, particularly for highly skilled workforces such as healthcare.

Evaluating Workplace Safety and Health Interventions: A Participatory Workshop

Workshop leader: Ted Scharf, PhD, NIOSH, Cincinnati

Case study leaders: Konstantin P. Cigularov, PhD, Old Dominion University, Norfolk, VA.; Chris Cunningham, PhD,  University of Tennessee at Chattanooga; Dan Hartley, EdD, and Marilyn Ridenour, MPH, NIOSH, Morgantown, WV

Co-authors: Kathleen Kowalski-Trakofler, PhD, NIOSH, Pittsburgh;  Karen Gilmore, MPH., University of Texas Health Center at Tyler,  John “Val” Valosen, B.A., CDC, Decatur, GA.,  Elizabeth Kapeller, M.A.H.S.,  Texas Health and Human Services Commission, Austin,  Keith Bletzer, PhD, MPH., Border Health Foundation, Tempe, AZ,  Sherri Lukes, RDH, MS, Southern Illinois University, Carbondale, and  Pamela Kidd, R.N., PhD formerly: Arizona State University, deceased.

Workshop Length: 6 hours

This participatory workshop is designed for:

  1. new researchers in occupational health psychology interested in evaluations in applied settings,

  2. experienced researchers and trainers who are interested in adapting this introduction to evaluation methodology to present to their clients, students or enhance their own research.

  3. business owners or managers who want to assess work organization changes,

  4. union and OSHA certified trainers who want to measure the effectiveness of their programs,

  5. healthcare workers interested in assessing client services, and

  6. environmental health and safety officers in industry thinking about how to measure the effectiveness of changes in workplace systems, processes, or equipment regarding worker safety and health.

Migrant health clinics, cooperative extension programs in agricultural safety and health, construction and mine safety programs, community-based health and social service programs, and emergency response activities, all may be viewed as natural, on-going experiments.  The health care, social services, and safety training that are routinely provided may be viewed as interventions for individuals, families, and whole communities, and for workers, work crews, and supervisors.  The only difference between these programs and a research or evaluation study is that systematic measurement is added to the provision of services.  The result of such measurement is a study of intervention effectiveness.  The authors take the position that all workplace safety, health, and social service interventions are amenable to systematic study in some fashion.  It is up to the safety or health care specialist to determine the needs in their workplaces and communities for:

  1. investigating new or puzzling questions,

  2. developing tools or procedures to respond to problems, and

  3. evaluating the effectiveness of new or continuing programs, tools, and procedures.

This participatory workshop will begin with typical problems (case studies) encountered in promoting and maintaining workplace safety and health and community health, for example:

1)  Enhancing Safety Climate through Leadership:  Reducing occupational accidents and injuries in the construction industry remains a high priority.  The Occupational Safety and Health Administration has stated that “developing strong safety cultures have the single greatest impact on accident reduction of any process.”  Furthermore, leadership plays a critical role in creating and enhancing organizational safety climates and cultures.  Therefore, an 8-week (2 hours per week) leadership training program aiming to improve safety leadership skills (e.g., emotion management, communication, influencing others) of senior apprentices/ journeymen/foremen was developed and conducted in three U.S. regions – Denver (CO), Portland (OR), and Chicago (IL), in partnership with the United Association: Union of Plumbers, Fitters, Welders, and HVAC Service Techs, and with the Mechanical Contractors Associations.  Each weekly session of the program contained three key components: active learning (e.g., simulations, role plays, case studies, and discussions), self-assessment (e.g., identifying strengths and challenges), and development (e.g., goal setting and practice skills on the job).  Workshop participants are challenged to design an evaluation to assess the effectiveness of the safety leadership program.  More specifically, the evaluation should consider short-term (e.g., improving safety leadership skills), intermediate (e.g., improving safety climate and performance), and long-term (e.g., reducing occupational injuries and illnesses) outcomes.

2)  Reducing Burnout Among Healthcare Providers:  Caring for others is difficult work and the healthcare profession is notoriously challenging for the occupational health and wellbeing of its workers.  Healthcare providers must often work very difficult schedules, while also balancing personal life challenges and difficult patient cases on a regular basis.  This case study builds on an actual, on-going intervention and research project that is targeted at identifying risk and reducing the likelihood of developing burnout among healthcare providers.  The intervention is built upon the tenets of Social Cognitive Theory, and is designed to increase residents’ self-efficacy regarding their abilities to manage and prevent work-related strain that can turn into burnout if chronically present.

3)  Violence Prevention in Veterans’ Hospitals: Psychiatric nurses report among the highest levels of violence for all nursing specialties.  There is a need, therefore, for interventions that effectively reduce violent occurrences on in-patient psychiatric units.  Currently, NIOSH and the Veterans’ Health Administration are working on a collaborative project to compare the effectiveness of three treatment programs.  Specifically, the effectiveness of a new treatment program—the Violence Prevention Community Meeting (VPCM)—will be compared to the effectiveness of Intensive Traditional Community Meetings (ITCM) and Treatment as Usual (TAU) meetings.  Impact of community meetings can be performed with measures related to the number and type of events, but improvements to the overall climate or culture of the workplace is harder to measure.

Workshop participants will select the intervention most relevant to them.  Next, in small groups (grouped by case study), participants will be challenged to develop a comprehensive research design to evaluate the effectiveness of the intervention.  Brief instruction will assist participants to:

  1. develop testable hypotheses, using both qualitative and quantitative approaches,

  2. build a research team that represents the entire community,

  3. incorporate strong ethical standards that are sensitive to local concerns,

  4. develop a research/evaluation design,

  5. recruit, select, and assign participants to study conditions,

  6. anticipate and plan for potential problems of validity,

  7. select appropriate measures that maximize reliability, and

  8. consider the effect sizes of their dependent measures and other features of a power analysis.

Each small group will then apply the instructional material, step-by-step, to the particular intervention (case study) they have selected.  At each step along the way, the small groups will report on their progress and receive feedback in a free-ranging discussion.  Four assignments will assist participants to construct a comprehensive research design to evaluate the case they selected:

  1. develop a model (specification of constructs and testable hypotheses),

  2. create a research design, including the timing of the intervention(s) and the appropriate pre- and post-test observations, as well as identifying potential comparison groups,

  3. identify the population and sample, the selection frame, and the assignment of participants to study conditions, and

  4. decide how to measure the selected constructs (identified in the model).

Learning objectives:  The goal of this workshop is to provide participants with an introduction to the tools necessary to think about their own problems, programs, and services in a systematic way.  The workshop will address:

  1. how to examine a problem and to suggest (hypothesize) potential causal relationships,

  2. how to design an intervention to measure the suspected causal factors and presumed intervention effects, and

  3. how to evaluate the effectiveness of the intervention.


This workshop assists participants to apply basic research methods to case study examples relevant to them.  Experienced researchers who want to adapt this workshop for presentation in their own communities are encouraged to contact the workshop organizers regarding the specific concerns of their client/student population and possible topics for relevant case studies.

CORRESPONDING AUTHORTed Scharf, PhD, Work Organization and Stress Research, National Institute for Occupational Safety and Health, 4676 Columbia Parkway, ms/C-24, Cincinnati, OH  45226, USA,  513-533-8170