Assessment Cyberguide for Learning Goals and Outcomes



	(TOPSS) Teachers of Psychology in Secondary Schools (PT@CC) Psychology Teachers at Community Colleges

	Reports

APPLYING ASSESSMENT STRATEGIES IN PSYCHOLOGY

Critique of Assessment Strategies
Applied to Goals and Outcomes

CHOOSING ASSESSMENT METHODS

Goal 1: Knowledge Base of Psychology

Demonstrate familiarity with the major concepts, theoretical perspectives, empirical findings, and historical trends in psychology.

1.1:	Characterize the nature of psychology as a discipline.
1.2:	Demonstrate knowledge and understanding representing appropriate breadth and depth in selected content areas of psychology: theory and research representing general domains, the history of psychology, relevant levels of analysis, overarching themes, and relevant ethical issues.
1.3:	Use the concepts, language, and major theories of the discipline to account for psychological phenomena.
1.4:	Explain major perspectives of psychology (e.g., behavioral, biological, cognitive, evolutionary, humanistic, psychodynamic, and sociocultural).

ASSESSMENT CATEGORY

ESTIMATE OF METHOD'S POTENTIAL

CLASSROOM/COURSE DATA	Overall Strong potential. Classroom activities and course data naturally provide venues in which to assess content knowledge in psychology. However, current assessment trends suggest that concentrating solely on these indices may not provide sufficient information to provide meaningful feedback on program integrity.
OBJECTIVE TESTS	Strong potential. Despite the inherent difficulties in constructing valid and reliable objective tests, the method can effectively assess content learning. However, most objective tests tend to evaluate student knowledge more routinely at lower levels of thinking (e.g., rote, simple application).
ESSAY TESTS	Strong potential. Despite the labor intensiveness of providing feedback on essay tests, this approach facilitates greater access to measuring deeper levels of content learning. Faculty are increasingly turning to the use of rubrics and specified criteria to address problems of reliability in grading.
EMBEDDED QUESTIONS AND ASSIGNMENTS	Strong potential. Choosing some courses in which program assessment activities can occur can still facilitate specific assessment of content knowledge. For example, embedding a departmental assessment of ability to demonstrate APA format in a methods class provides a a reasonable vehicle for assessing content knowledge of APA format. Other emphases are possible in embedded assessments, including ethics, persistent themes, or historical detail among others.
CLASSROOM ASSESSMENT TECHNIQUES	Strong potential. Faculty engage in classroom assessment techniques to provide spot checks of how well students are learning specific concepts. Although the focus is understandably narrow (e.g., the content of a particular class), the method provides optimal feedback for the faculty member concerned with what students are learning and retaining.

INDIVIDUAL PROJECTS/PERFORMANCE ASSESSMENT	Overall Strong potential. The knowledge base of psychology is predictably the foundation for most individual projects. These generally offer the advantage of studying some corner of the discipline in depth; however, breadth of exposure to content may be a casualty given the time limitations most faculty and students face.
WRITTEN PRODUCTS	Strong potential. When assignments are well-designed, written products should provide insight into what students know and don't know about content. Faculty have discovered that specifying how much content (e.g., number of required references) may facilitate the depth of exploration the faculty member had in mind when designing the project instructions.
ORAL PRESENTATIONS	Strong potential. oral presentations also provide insight into student learning of the content. In addition, the opportunity to engage students in questions allows faculty and classmates to probe the depth of student knowledge while building oral communications skills.
GRAPHIC TESTS AND DISPLAYS	Strong potential. Concept maps can reveal the nature of associations that students develop regarding specified content in the discipline.
POSTERS	Strong potential. posters can provide a more global sense of what students understand due to the brevity of the medium. However, informal questioning can fill in the gaps about what students have not communicated in the poster.
STRUCTURAL/SITUATIONAL ASSESSMENTS	Strong potential. Although situational assessments tend to emphasize application of learning, applications are built on a disciplinary foundation. The success of assessment of content learning will depend on the expertise demonstrated in the design of the applied assessment.

SUMMATIVE PERFORMANCE ASSESSMENT	Overall Mixed potential. Assessments that occur at the end of a program vary in their effectiveness for assessing content. In some cases, depth of knowledge required by some demonstrations will not allow an estimate of broad knowledge in the discipline.
STANDARDIZED TESTS	Mixed potential. Currently there are two primary standardized exams that allow for comparison across institutions as well as tracking changes in program achievement over time: the Academic Concentration Applied Test (ACAT) and the Major Fields Test by ETS. Each exam measures knowledge in the subdisciplines of psychology, but student course selection may adversely affect overall performance on either instrument. Care must be exercised in interpreting the results.
LOCALLY DEVELOPED TESTS	Strong potential. Developing a department examination is a time-consuming but effective way to track changes in student knowledge over time but does not provide normative comparison with other programs. In addition, test security and changes in content knowledge make this practice complex.
CAPSTONE EXPERIENCES	Mixed potential. The utility of capstone performance to assess content knowledge greatly depends on the scope of the course design. Students are more likely to develop deep levels of expertise in more narrowly defined areas of psychology in most capstone designs. To the extent that their performance represents what they can do within specific performance parameters, the capstone may be a satisfying method to assess the ability to deal with content in sophisticated ways. However, more broadly conceived capstone courses (e.g., history, systems of psychology) may provide broader assessment opportunities.
INTERNSHIP/PROFESSIONAL APPLICATIONS	Moderate potential. Internships and professional applications facilitate specific types of applications. For example, an industrial-organizational internship may be an optimal way for a student to demonstrate the knowledge base related to the subdiscipline, but it may not be satisfying as a broad assessment.
PORTFOLIOS	Strong potential. Selecting and justifying selections from explicit departmental criteria will facilitate student reflection regarding the level of expertise they have developed in the content of psychology.
ASSESSMENT CENTER METHODS	Strong potential. Similar to structured assessments, the in-basket strategies of assessment center methods can provide insight into student abilities to apply principles from the content of psychology.
CASE AND LONGITUDINAL STUDIES	Limited potential. Because much of content learning is fragile, longitudinal studies of content retention are likely to be disappointing sources of student learning. In addition, merely reporting how sturdy content learning is over time rather than directly assessing may be content learning will be a less reliable measure.

SELF-ASSESSMENT	Overall Mixed potential. As can be seen from student anticipation of how well they performed on an exam, student ability to judge their own expertise is variable.
STUDENT JOURNALS	Moderate potential. Journal instructions can specify the manner in which students should interact with the content of the discipline. For example, instructions might require that students demonstrate the appropriate application of five concepts or principles from the discipline. Students will vary in their own expert judgment on the success of addressing the concepts or principles in the manner anticipated by the faculty.
SELF-CRITIQUES	Mixed potential. Students have limited experience in making judgments about how well they have met the content criteria of a given assignment. Students often drift to the easier-to-judge aspects of performance, such a format concerns, interest generation, or comfort level rather than exploring how well they have reflected content expertise.

COLLABORATION	Mixed potential. Some methods offer effective avenues for examining content and theory, while others are less promising.
RESEARCH TEAMS	Strong potential. Research teams give students an opportunity to demonstrate content expertise in two dimensions: research methods and the subject matter that the research is designed to explore. Students can receive separate evaluations on the extent to which they have collectively demonstrated research expertise as well as whether they have appropriately represented the target content.
GROUP PROJECTS	Strong potential. Group projects can also provide a content-based opportunity to develop group skills. Projects can reflect successful or unsuccessful strategies to master relevant content and principles. However, group projects suffer similar limitations to individual projects. Committing in-depth study to one arena may require the sacrifice of exposure to other content in the course.
ON-LINE ACTIVITIES	Limited potential. Unless students are given very constrained instructions regarding how to pursue content collaboration, the use of chat room or email exchanges to monitor content expertise may be challenging.

INTERVIEWS & SURVEYS	Mixed potential. In general, surveys and interviews are not recommended because the assessment of content is not likely to be direct.
SATISFACTION SURVEYS	Poor potential. Assessing content expertise through satisfaction surveys is too indirect to be recommended.
PERFORMANCE REVIEWS	Moderate potential. Although this method is time consuming, the next step (e.g., grad school or employment) can provide for direct observation of the content of psychology.
EXIT INTERVIEWS	Limited potential. Exit interviews tend to focus on affective dimensions of learning as well as the collection of impressions that may facilitate program improvement. Content mastery is not routinely the focus of exit interviews.
EXTERNAL EXAMINER INTERVIEWS	Moderate potential. A rigorous external examiner protocol could focus on the depth of content mastery of individual students. However, the expense and time limitations of this approach tend to focus on other aspects of student performance.
FOCUS GROUPS	Not recommended. Focus groups typically convene to solve a specific problem rather than provide a measure of content mastery. Such academic development may be inferred but there are other more direct methods to assess mastery.
FOLLOW-UP ALUMNI INTERVIEWS	Not recommended. Engaging with alumni over the specifics of content that they can recall is likely to be a discouraging assessment strategy since the detail of the discipline dims with distance from graduation.

ARCHIVAL MEASURES	Mixed potential. Archival measures can provide some insight into the content bases to which students have been exposed but will do little to assess more formal learning of the content in the discipline.
TRANSCRIPT ANALYSIS	Limited potential. Individual transcript analysis can provide not just a measure of the various content bases to which the student has been exposed but through grades can provide a gross measure of achievement in those areas. However, assessment experts recommend that other noncourse-based strategies will be more effective in providing legitimate measures of student and program achievement.
ANALYSIS OF TRANSFER PATTERNS	Limited potential. Examining patterns of what transfer students provide can help departments determine what and when to offer in the curriculum, but will shed little light on the quality of learning.
SYLLABUS AUDIT	Limited potential. A syllabus audit can isolate the range of content exposure that students experience but will be poor indicators of actual learning.
DEMOGRAPHIC DATA ANALYSIS	Not recommended. Understanding the characteristics of the student body will provide little insight into their content mastery.
ALUMNI DATABASE	Not recommended. The alumni database does not directly reveal student expertise in the content of psychology. However, many programs rely on the percentage of students who go on to graduate school in the area as an indirect measure of content expertise.
LIBRARY STATISTICS USAGE/WEB HITS	Not recommended. Content expertise is not apparent in this archival analysis.

CHOOSING ASSESSMENT METHODS

Goal 2: Research Methods in Psychology

Understand and apply basic research methods in psychology, including research design, data analysis, and interpretation.

2.1	Describe the basic characteristics of the science of psychology.
2.2	Explain different research methods used by psychologists.
	Describe how various research designs address different types of questions and hypotheses Articulate strengths and limitations of various research designs Distinguish the nature of designs that permit causal inferences from those that do not
2.3	Evaluate the appropriateness of conclusions derived from psychological research.
	Interpret basic statistical results Distinguish between statistical significance and practical significance Describe effect size and confidence intervals Evaluate the validity of conclusions presented in research reports
2.4	Design and conduct basic studies to address psychological questions using appropriate research methods.
	Locate and use relevant databases, research, and theory to plan, conduct, and interpret results of research studies Formulate testable research hypotheses, based on operational definitions of variables Select and apply appropriate methods to maximize internal and external validity and reduce the plausibility of alternative explanations Collect, analyze, interpret, and report data using appropriate statistical strategies to address different types of research questions and hypotheses Recognize that theoretical and sociocultural contexts as well as personal biases may shape research questions, design, data collection, analysis, and interpretation
2.5	Follow the APA Code of Ethics in the treatment of human and nonhuman participants in the design, data collection, interpretation, and reporting of psychological research.
2.6	Generalize research conclusions appropriately based on the parameters of particular research methods.
	Exercise caution in predicting behavior based on limitations of single studies Recognize the limitations of applying normative conclusions to individuals Acknowledge that research results may have unanticipated societal consequences Recognize that individual differences and sociocultural contexts may influence the applicability of research findings

ASSESSMENT CATEGORY

ESTIMATE OF METHOD'S POTENTIAL

CLASSROOM/COURSE DATA	Mixed potential. Useful for providing assessment of factual knowledge and some limited application. Research skills will be better assessed with other strategies that involve activities outside the traditional classroom.
OBJECTIVE TESTS	Moderate potential. Good method for assessing primarily factual knowledge, especially if the test questions can be demonstrated to have strong psychometric properties. Less useful in assessing application and higher level comprehension such as designing original research or performing and interpreting statistical calculations.
ESSAY TESTS	Moderate potential. More powerful method for assessing application and higher level comprehension, but is still limited in its ability to assess ability to design original research or perform and interpret statistical calculations because of limited time frame in testing situation.
EMBEDDED QUESTIONS AND ASSIGNMENTS	Strong potential. Because research methods courses are often a prerequisite for advanced courses, departments may also dictate specific assignments (e.g., research projects) that should be embedded in required coursework across different sections of the same methods and/or statistics courses and in subsequent courses in the curriculum.
CLASSROOM ASSESSMENT TECHNIQUES	Moderate potential. Provides quick, but often limited assessment, on student understanding and performance.

INDIVIDUAL PROJECTS/PERFORMANCE ASSESSMENT	Strong potential. Faculty can interpret sophistication in research skills from intellectual products. Because projects are done outside class, there may be some risk that a student's work is not an individual production. However, this limitation can be addressed by including a reflective piece that assesses the targeted skills.
WRITTEN PRODUCTS	Strong potential. Written reports of original research provide an ideal context for assessing the methodological skills and ethical issues involved in designing, conducting, and evaluating the results. Time constraints (e.g., IRB approval) and the labor intensive nature of original research may limit usefulness in some courses. Research projects may also only assess an understanding of the particular methods used, not a broader understanding.
ORAL PRESENTATIONS	Moderate potential. Individual oral presentations provide some opportunity to evaluate quality of research skills and ability to present a shorthand summary. However, these may be challenging to judge in the moment and they often lack details that allow for in depth assessment.
GRAPHIC TESTS AND DISPLAYS	Moderate potential. Graphical presentations can be useful in mapping the research process. Statistical understanding can be assessed through the accuracy and clarity of graphical presentations.
POSTERS	Moderate potential. The limited space available in most posters may not provide an ideal context in which to evaluate the full understanding and application of research methods.
STRUCTURAL/SITUATIONAL ASSESSMENTS	Strong potential. The presentation and interpretation of research findings in the popular media can be used to have students demonstrate their skill in addressing issues related to the design and interpretation of research. Current events can also be used as a starting point for students to design and conduct original research projects.

SUMMATIVE PERFORMANCE ASSESSMENT	Mixed potential. Strategies in this category range from poor to strong.
STANDARDIZED TESTS	Moderate potential. Although standardized tests assess factual knowledge related to research methods and statistics, they fail to evaluate application of skills at the level identified for these outcomes.
LOCALLY DEVELOPED TESTS	Moderate potential. Like standardized tests, they primarily focus on factual knowledge as opposed to application. In addition, they may lack strong psychometric properties.
CAPSTONE EXPERIENCES	Strong potential. Assuming that the capstone course or project has an expressive requirement (e.g., writing or speaking), it can provide an integrated demonstration opportunity.
INTERNSHIP/PROFESSIONAL APPLICATIONS	Limited potential. The focus of most applied internship experiences tends to be in applied areas of psychology. However, some experiences may include the opportunity to design, conduct, and evaluate research (i.e., a research internship).
PORTFOLIOS	Strong potential. Explicit criteria that ask students to select "works" based on what these reveal about their research skills can provide an opportunity to evaluate the evolution of their abilities through a focused reflection on why they selected the items they did.
ASSESSMENT CENTER METHODS	Poor potential. Assessment center methods are generally limited in time and focus. They are unlikely to provide in depth information on all the outcomes associated with this goal because of inherent time constraints.
CASE AND LONGITUDINAL STUDIES	Limited potential. The complexities of tracking all of these skills over time mitigates against adopting this strategy to monitor their evolution, especially where different curses in the curriculum vary in the requirement to use these skills. Programs requiring both a survey research methods course with an original research project and a capstone research experience may have a limited opportunity to evaluate longitudinal development.

SELF-ASSESSMENT	Mixed potential. Developing students' metacognition of their understanding of research methods has seldom been addressed self-assessment strategies.
STUDENT JOURNALS	Moderate potential. Although it may be unlikely that faculty would choose to invest time reading about students' struggle to learn research methods, this technique can be adapted to a research journal where students keep a record of research ideas, development and progress that reflects application of research methods knowledge.
SELF-CRITIQUES	Limited potential. When faculty can specify relevant performance criteria, students can provide an estimate of their research and statistical skills.

COLLABORATION	Moderate potential. Techniques in this category are moderate to limited in usefulness.
RESEARCH TEAMS	Moderate potential. Research teams can develop and evidence expertise in research skills through peer involvement and often model the collaborative nature of research at the professional level. Unfortunately, research teams may reduce a beginning or weak student's direct involvement in generating research ideas, research design, statistical analysis, and interpretation of results.
GROUP PROJECTS	Moderate potential. Group projects involve similar issues to those of research teams.
ON-LINE ACTIVITIES	Limited potential. Archived on-line chat rooms, listservs, or bulletin boards can provide opportunities to assess the development and evolution of research ideas from start to finish.

INTERVIEWS & SURVEYS	Mixed potential. The assessment of attitudes by the students or other stakeholders may provide some feedback about research methods and statistical competence, but attitudes may not be an accurate indication of true skill attainment.
SATISFACTION SURVEYS	Strong potential. The abilities of students to design, conduct and evaluate research can be evaluated by employers, graduate advisors, or other stakeholders. External evaluators may explicitly need to be prompted to address these skills. This may be particularly effective for those students who continue in graduate programs in psychology.
PERFORMANCE REVIEWS	Strong potential. Stakeholders can provide an estimate of strengths and weaknesses within research skills with appropriate prompts for reflection.
EXIT INTERVIEWS	Moderate potential. Students can be asked to reflect on the evolution of their research and statistical skills.
EXTERNAL EXAMINER INTERVIEWS	Limited potential. External examiners can ask probe questions to evaluate student comfort levels about research and statistical skills, but the evaluation of self-report relative to actual performance quality may be problematic unless evaluators also review actual products.
FOCUS GROUPS	Limited potential. Although focus groups most often convene to solve specific departmental problems, this area is often core to a program and challenging to students and may be more likely to be addressed in this context.
FOLLOW-UP ALUMNI INTERVIEWS	Limited potential. Although a follow-up interview on this topic might invite demand characteristics, it may be useful to assess perceived skill levels in post-graduate settings.

ARCHIVAL MEASURES	Limited potential.
TRANSCRIPT ANALYSIS	Moderate potential. Transcript analysis can reveal the pattern of courses students may engage in (or avoid) in the development and use of research method s and statistical skills. The transcript analysis can provide both patterns and some in-class estimates of quality of student performance although the value of these may be limited.
ANALYSIS OF TRANSFER PATTERNS	Limited potential. Departments can benefit by understanding the transfer courses that students may have taken in research methods and statistics and making comparisons to students who took departmental courses in these areas.
SYLLABUS AUDIT	Moderate potential. An analysis of which courses include content or projects emphasizing research methods or statistics may be a helpful first step in diagnosing where these skills need to be enhanced.
DEMOGRAPHIC DATA ANALYSIS	Not applicable.
ALUMNI DATABASE	Not applicable.
LIBRARY STATISTICS USAGE/WEB HITS	Not applicable.

CHOOSING ASSESSMENT METHODS

Goal 3: Critical Thinking Skills in Psychology

Respect and use critical and creative thinking, skeptical inquiry, and, when possible, the scientific approach to solve problems related to behavior and mental processes.

3.1	Use critical thinking effectively.
3.2	Engage in creative thinking.
3.3	Use reasoning to recognize, develop, defend, and criticize arguments and other persuasive appeals.
3.4	Approach problems effectively.

ASSESSMENT CATEGORY

ESTIMATE OF METHOD'S POTENTIAL

CLASSROOM/COURSE DATA	Overall Mixed potential. Classroom and course data can be used to assess critical and creative thinking, but the quality of the assessment depends on what is measured in these settings and not the setting per se.
OBJECTIVE TESTS	Limited potential. Objective tests can be used to assess critical thinking, but good objective tests of critical thinking are difficult to construct and cannot be used to assess what students actually do in an unstructured setting where critical thinking is required. They cannot assess the propensity to engage in critical thought. They are better as measures of recognition memory, and hence of limited usefulness in assessing critical thinking.
ESSAY TESTS	Strong potential. An essay test that poses an ecologically-valid scenario (ideally somewhat complex) where students need to explain/ describe their thinking and the conclusion they reached or problem they solved can be a good way to assess critical thinking.
EMBEDDED QUESTIONS AND ASSIGNMENTS	Strong potential. An embedded question or assignment can provide a measure of student's propensity to think critically (i.e., do they engage in critical thinking when the need for critical thinking is not cued or labeled).
CLASSROOM ASSESSMENT TECHNIQUES	Poor potential. Classroom assessment techniques can include reflections on what was learned. It is more likely useful as feedback to instructors about what students believe they have learned than a measure of learning per se.

INDIVIDUAL PROJECTS/PERFORMANCE ASSESSMENT	Overall good potential. Individual projects and performance assessment can be good measures of critical thinking, assuming that the project calls for extended and careful thought. The nature of the project or performance (e.g., solve a novel problem) is what determines the quality of the assessment. The quality of a critical thinking assessment most often lies in the way the instructor crafted the assignment and explained it to students.
WRITTEN PRODUCTS	Strong potential. Like essay tests, a written project needs to allow the student to show the thinking process that went into a conclusion or a solution to a problem.
ORAL PRESENTATIONS	Strong potential. oral presentations are just an alternative format for presenting one's thinking, and thus are similar to written products in their ability to assess critical thinking.
GRAPHIC TESTS AND DISPLAYS	Strong potential. A graphic display of one's thinking can be an excellent assessment of the quality of a student's thinking. There are many concept maps and other ways to map verbal information onto spatial arrays that are well suited for critical thinking assessment. A completed template that shows the parts of a persuasive argument, for example, can be used to clarify complex topics and provide a "picture" of the student's thinking.
POSTERS	Strong potential. A poster can, and probably should, contain a mix of verbal and graphic displays. It can be used to assess critical thinking, if the topic or reason for the poster requires critical thought.
STRUCTURAL/SITUATIONAL ASSESSMENTS	Strong potential. When this category includes activities such as role-playing, seeing problems from multiple perspectives, and similar activities, it can be a good way to demonstrate critical thinking skills.

SUMMATIVE PERFORMANCE ASSESSMENT	Overall Mixed potential. Summative assessments usually refer to tests that are normed to provide comparative data, usually at the completion of a program of study. The normative information can be useful, but only if the test is valid in that it relates to the way students think critically when they are not in class.
STANDARDIZED TESTS	Moderate potential. There are several standardized tests of critical thinking. Unfortunately, they tend to be multiple-choice tests with short problems or scenarios described in each question. They are not generally good measures because real life is much messier, and there is rarely a single correct answer to ill-defined problems. They also do not measure what student's actually do in less structured settings. A quality standardized test is possible, if it includes both constructed response and multiple-choice alternatives to show how students approach problems and whether they can recognize a good response when they have to select from among a small set of alternatives.
LOCALLY DEVELOPED TESTS	Limited potential. A locally developed test can reflect the curriculum that is taught, so it can be more useful to instructors, but locally-developed tests will rarely have the psychometric properties of good reliability and validity that a standardized test will have.
CAPSTONE EXPERIENCES	Strong potential. A capstone experience can be an extended project that requires the application of critical thinking skills to a wide range of issues. If well designed, capstone experiences can provide meaningful measures of critical thinking, but the instructor needs to have clear critical thinking objectives in mind when planning the capstone experience because it is not likely to assess critical thinking without deliberate planning.
INTERNSHIP/PROFESSIONAL APPLICATIONS	Limited potential. Ideally, an internship or other professional application would require evidence of critical thinking, but they would need to be designed with this specific outcome in mind and that rarely happens. In general, instructors do not have direct control over internship experiences, which can make the value of an internship or other application as an assessment of critical thinking hit-or-miss.
PORTFOLIOS	Strong potential. A portfolio that is well planned to show growth in critical thinking skills over time (e.g., four years in college) can provide a good index of gains in critical thinking. Like the other methods, the value of portfolios depends on how carefully the intended critical thinking outcomes are articulated and carried across several different courses.
ASSESSMENT CENTER METHODS	Strong potential . An assessment center is usually a single place where a variety of assessment activities are planned and data are collected. Often they will include simulations of real-life scenarios and problems. If the activities are well planned, they can provide valuable data about critical thinking (e.g., an in-basket exercise).
CASE AND LONGITUDINAL STUDIES	Limited potential. Following students' development in critical thinking over time may be somewhat challenging in the absence of objective measures of that growth.

SELF-ASSESSMENT	Overall Mixed potential. A self-assessment is a student's own analysis of how well he or she is thinking. We know from a large number of studies that most people are poor judges of how well or how critically they think. These techniques can be useful over time if students learn to be more accurate in their self-assessment, but there are little data to suggest that this actually happens.
STUDENT JOURNALS	Mixed potential. Student journals seem to work well for a subset of students who actually reflect on their thinking. For many students, they are worthless exercises in filling up paper. Some instructors like to give students the experience of student journals so that those students who benefit from this activity are not penalized by those who do not. It can be useful, but only for some students. Clear instructions for journaling are required.
SELF-CRITIQUES	Limited potential. Given that most people are very happy with their ability to think critically, self-critiques can only be useful if students learn to see their own weaknesses. It is difficult to change belief about how well one thinks, but not impossible. Thus, one outcome of critical thinking instruction is the seemingly paradoxical result that students often rate themselves as poorer thinkers at the end of a course than at the start. This is a positive outcome, but it tells instructors very little about the student's actual ability to think critically.

COLLABORATION	Overall Strong potential. When students collaborate, they think in groups. Because much of the thinking they will do outside of class will involve other people, it can be a valid approach to assess critical thinking.
RESEARCH TEAMS	Strong potential. If the research team is given a problem that requires critical thinking and good guidelines for teamwork are provided so that each team member must contribute to some of the thinking, it can be useful. Instructors will want to capture at least a sample of the group thinking process so that it can be reviewed with each team.
GROUP PROJECTS	Strong potential. Group projects may be designed so that success only can occur when the group engages in effective critical thinking. Groups can process where their critical thinking was faulty to learn from their error.
ON-LINE ACTIVITIES	Limited potential. An on-line collaboration offers the possibility of tracking the thinking process via the written exchanges among team members. Of course, instructors would want students to know that their exchanges are being monitored.

INTERVIEWS & SURVEYS	Overall Limited potential. Interviews and surveys usually ask specific questions about individual beliefs and perceptions. They are not useful in assessing what is learned because they to focus on what students believe they learned and how satisfied they are with the learning.
SATISFACTION SURVEYS	Poor potential. Satisfaction surveys are often called "smilies" because respondents indicate how happy they are with an assignment or course. These are not the same as actual measures of what was learned and cannot be substituted for performance indicators.
PERFORMANCE REVIEWS	Limited potential. Stakeholders can be asked to rate the quality of critical thinking in their evaluations of performance but may require training to understand the parameters being investigated.
EXIT INTERVIEWS	Poor potential. Exit interviews occur when students are leaving a course or program, most often at graduation. They are reflections about what was good or bad about a program of study. Although these measures provide useful data, they usually do not measure critical thinking.
EXTERNAL EXAMINER INTERVIEWS	Moderate potential. External examiners are used to provide an outside (i.e., unbiased) evaluation of the quality of learning. The ability of external evaluators to measure critical thinking depends on what they ask. If they ask satisfaction questions, then they are not assessing critical thinking, but if they ask students to think through a complex problem and explain what they are doing, the assessment can be a measure of critical thinking.
FOCUS GROUPS	Limited potential. Focus groups are often group evaluations of a program or course. They do not provide evidence of critical thinking unless the group is asked to solve a problem, reach a conclusion, make a complex decision or engage in some other critical thinking task.
FOLLOW-UP ALUMNI INTERVIEWS	Moderate potential. Alumni follow-ups tend to utilize satisfaction questions, but they could provide evidence of the long-term retention of critical thinking skills and their transfer to novel domains if the alumni are asked questions that require critical thinking.

ARCHIVAL MEASURES	Overall Poor potential. Archival methods use data that are already available. As in the other categories, the quality of the assessment depends on what is in the available data.
TRANSCRIPT ANALYSIS	Limited potential. A transcript analysis is not likely to tell us much about critical thinking skills because we do not know what was required in each of the classes. Research has shown that much of the learning that occurs in college is relatively low level direct recall of information or low-level inferences.
ANALYSIS OF TRANSFER PATTERNS	Limited potential. Course-taking patterns are not likely to useful by themselves, but could be useful to see how different patterns relate to more valid measures of critical thinking.
SYLLABUS AUDIT	Mixed potential. An audit of a syllabus can sometimes show if critical thinking skills are being taught and learned in a particular class, but most often the syllabus is a list of reading assignments, dates assignments are due, and exam dates. There is rarely any information in the syllabus that provides a clue as to what students are required to do with the information to-be-learned.
DEMOGRAPHIC DATA ANALYSIS	Poor potential. These analyses will not tell us anything about the quality of the thinking of any individual or group.
ALUMNI DATABASE	Poor potential. An alumni data base that provides information about the types of careers that students enter upon graduation and where they are working in 5 to 10 years can allow us make inferences about critical thinking. In general, some careers (e.g., law, research) require better thinking skills than others (e.g., most clerical positions), but this is not a strong assessment method.
LIBRARY STATISTICS USAGE/WEB HITS	Poor potential. Ideally, students who read more should be better thinkers, but we do not know if this hypothesized relationship is true. A better index might be what they chose to read, but this is not a direct measure of critical thinking ability and it requires too many inferences to qualify as a valid assessment.

CHOOSING ASSESSMENT METHODS

Goal 4: Application of Psychology

Understand and apply psychological principles to personal, social, and organizational issues.

4.1	Describe major applied areas of psychology (e.g., clinical, counseling, industrial/organizational, school, health).
4.2	Identify appropriate applications of psychology in solving problems, such as
4.3	Articulate how psychological principles can be used to explain social issues and inform public policy.
4.4	Apply psychological concepts, theories, and research findings as these relate to everyday life.
4.5	Recognize that ethically complex situations can develop in the application of psychological principles.

ASSESSMENT CATEGORY

ESTIMATE OF METHOD'S POTENTIAL

CLASSROOM/COURSE DATA	Mixed potential. Classroom strategies show variable potential in measuring how students apply the concepts and principles they learn in their psychology courses.
OBJECTIVE TESTS	Limited potential. objective tests (e.g., multiple-choice items) can assess knowledge of the roles of applied areas (e.g., employee selection, training, and evaluation in I/O psychology) and the differences among areas of applied psychology (e.g., clinical and counseling psychology).
ESSAY TESTS	Strong potential. Essay questions can assess knowledge of the application of psychology if they require students to describe examples of how psychological principles and methods can be used to solve specific problems (e.g., decreasing a child's tantrums, strengthening a college student's study skills, or helping an adult overcome a phobia) or how ethical issues can decrease the desirability of some applications.
EMBEDDED QUESTIONS AND ASSIGNMENTS	Strong potential. Embedded assessments represent activities that the department has agreed will provide a good measure of student learning or progress but take place within the confines of a class. Classroom assignments can be used to assess students ability to apply psychological principles, theories, and methods if they are designed to do so. For example, students can apply what they have learned about stress management in an assignment that requires them to (1) identify the major stressors in their lives, (2) devise a plan to improve their ability to cope with these specific stressors, and (3) evaluate the effectiveness of their plan.
CLASSROOM ASSESSMENT TECHNIQUES	Not applicable. Classroom assessment refers to informal methods to determine whether or not students understand course material (e.g., the end-of-class one minute paper). Thus they are more suited to providing feedback to teachers about what is going on in their classrooms than producing data about students ultimate ability to apply psychological principles and methods.

INDIVIDUAL PROJECTS/PERFORMANCE ASSESSMENT	Mostly Strong potential. Nearly all of the methods that address individual performance are reasonable to optimal means of addressing the application of concepts.
WRITTEN PRODUCTS	Strong potential. Any written assignment (e.g., a term paper, moral dilemma analysis) that requires students to describe how they would apply their psychological knowledge would be useful as a means of assessing knowledge of how psychological principles and methods can be applied. Of course, knowing how to apply psychological principles and methods and actually being able to apply them successfully are two different stories.
ORAL PRESENTATIONS	Strong potential. oral presentations can be used to assess the ability to apply psychology in the same way that written products can.
GRAPHIC TESTS AND DISPLAYS	Limited potential. Simple graphic representations may be insufficient with regard to clarifying an application.
POSTERS	Strong potential. A poster can provide substantial evidence of students ability to apply what they have learned in their methods classes. It can also provide faculty with an opportunity to evaluate students ability to "think on their feet" when they are asked questions during a poster session, which provides another venue for demonstrating application skills.
STRUCTURAL/SITUATIONAL ASSESSMENTS	Strong potential. Simulations in which a "real world" situation is created in an artificial environment (e.g., counseling sessions in which the student "counselor" must provide counseling to a fellow student who is role playing a particular DSM category) can provide faculty with a rich opportunity to assess students ability to apply what they have learned in the classroom (e.g., listening skills, the development of rapport, professional mannerisms, etc.).

SUMMATIVE PERFORMANCE ASSESSMENT	Mixed potential. Summative methods must have a predominant focus on application to serve this goal. Some summative approaches tend to have a broader focus.
STANDARDIZED TESTS	Limited potential. There are some nationally standardized tests (e.g., the ACAT and the ETS Undergraduate Field Test) that contain subtests the measure student's knowledge of psychological research methods (e.g., research design, statistical analysis, and graphic interpretation), which are legitimate examples of how psychologists apply the scientific method to solvable problems. However, most question sets favor lower-order questions rather than those that involved applied skills.
LOCALLY DEVELOPED TESTS	Limited potential. A locally developed test will allow faculty the opportunity to collaborate to produce a locally developed test that incorporates application, but producing applied items will be as difficult locally as it is in national exams.
CAPSTONE EXPERIENCES	Optimal potential. There are many types of capstone experiences, but those the require enrollees to "do the discipline" are probably the most effective way for a department to assess its student's ability to apply the psychological principles and methods they have acquired in their previous class work. Capstone classes provide students with an opportunity "to demonstrate comprehensive learning in their major through some type of product or performance" (Palomba & Banta, 1999, p. 124). In other words, a capstone is a class in which senior psychology majors are required to pull together what they have learned in their previous classes and use this integrating experience to demonstrate they are capable of doing what they should be able to do as they graduate from the program (e.g., perform research in a capstone laboratory or demonstrate clinical skills during an internship with a local crisis clinic). This process serves a dual purpose. It allows psychology majors with a final opportunity to practice and demonstrate the skills they will need to succeed after graduation on the job or in graduate school. It also provides the Psychology Department with a final opportunity to assess whether or not it has been successful in its mission to produce psychology majors who are capable of applying what they have learned in their previous seven semesters.
INTERNSHIPS OR PROFESSIONAL APPLICATIONS	Strong potential. See Capstone Experiences above for description of internships as assessment strategies. An internship or practicum taken under the direction of an on-the-job professional can be an invaluable experience for psychology students and it can also provide quality feedback to a department about its students' ability to apply what they have learned in the classroom if their on-the-job supervisors are willing and able to provide such feedback to the supervising teacher.
PORTFOLIOS	Moderate potential. Portfolios can produce longitudinal information, allow students to reflect upon their progress, and give them a voice in assessment. Artifacts could include test scores in classes that covered application topics, papers written on application, journals from internships, reports of projects, etc. The degree to which application is involved in the portfolio design criteria must be departmentally determined.
ASSESSMENT CENTER METHODS	Mixed potential. Simulations are an excellent way to assess application skills, but the necessity for thorough planning and implementation, plus the expense of training or paying already-trained assessors are strong drawbacks of this method. It might be interesting to train senior psychology majors (as part of their capstone experience) to be assessors in simulations conducted in lower-level classes in which psychological principles, theories, and methods are applied (e.g., case studies requiring DSM diagnoses or detection of flaws such as uncontrolled variables in research designs). This would allow faculty to not only involve students in the assessment process, but also provide students with the opportunity to learn and demonstrate a valuable application of psychology (assessment).
CASE AND LONGITUDINAL STUDIES	Mixed potential. Longitudinal assessment studies involve the collection of pre- and post-information and, as such, they can provide evidence of how students change in their ability to apply the discipline over the course of their education. This type of assessment may be most valuable to departments in non-selective institutions whose students enter with minimal skills. These departments may seek to prove that although their students do not graduate with the same high level of skills exhibited by the graduates of more selective school, their students actually make more progress (i.e., more added value) during their undergraduate years than their more high ability counterparts. As with all types of longitudinal design, it is important to realize that pre- and post-changes may be due to factors other than academic programs (e.g., maturity), and that tracking students through the process can be challenging (e.g., students who drop out).

SELF-ASSESSMENT	Strong potential. Both self-assessment methods show promise for assessing application skills in psychology.
STUDENT JOURNALS	Strong potential. Students engaged in internships or service learning projects can be required to journal their experiences by keeping time logs, describing their actual activities, identifying their goals, evaluating whether or not they have accomplished their goals, and illustrating how their goals have been met. A perusal of these journals can give departments an accurate idea of their student's perceptions of their ability to apply the psychological principles and methods they have acquired in the classroom. However, better journal performance is facilitated by explicit directions to reflect application of course content.
SELF-CRITIQUES	Strong potential. When students have criteria that delineate successful performance, they can demonstrate the ability to judge their own skills in applying psychology concepts and principles.

COLLABORATION	Mixed potential. Traditional group projects and research teams show great potential for illustrating application skills; on-line tracking is much more problematic.
RESEARCH TEAMS	Strong potential. One specialized type of group problem-solving is the research team, which requires its members to apply what they have learned in their research methods class in a collaborative setting. The research team can empower students to learn how to apply methods to solve problems without having the full responsibility involved in solo projects. In addition, the number of team members has the potential to enhance the quality of the application just as it has the potential to make completing the project more challenging.
GROUP PROJECTS	Strong potential. Group projects allow faculty to assess their students ability to apply the principles they have acquired in two ways. When students work together to solve problems, they can demonstrate applications in content such as using Kohlberg's stages to determine moral reasoning. Their group work can also illustrate what principles from social psychology can be brought to bear to make the work satisfying (e.g., how to minimize social loafing). Applying these principles is an excellent example of the application of psychological knowledge to both everyday life (e.g., persuading children to do household chores rather than being waited upon by their mothers as if they were members of the royal family) and organizational situations (e.g., getting maximum performance from all members of a committee or work team). Carefully devised rubrics to assess collaboration attitudes and skills (e.g., willingness to volunteer and consensus-building) can be used by both faculty and peers at strategic stages of a project.
ON-LINE ACTIVITIES	Limited potential. Tracking group problem-solving process through online discussion can be a rich source of data for determining the evolution of application skills; however, the disadvantages involved in deconstructing the qualitative materials make this strategy less desirable.

INTERVIEWS & SURVEYS	Moderate potential. Interview methods generally can have application skills as a target but this strategy requires making the assessment of application skills a prominent part of the design.
SATISFACTION SURVEYS	Moderate potential. Satisfaction surveys can be used to determine how well current students or alumni perceive they are learning or learned how to apply psychology. However, the survey must be carefully crafted to reflect an estimate of the student's application skills.
PERFORMANCE REVIEWS	Strong potential. Subsequent work settings provide good contexts in which generalization of skills can be evaluated.
EXIT INTERVIEWS	Moderate potential. Exit interviews can be designed to focus on the aspects of application outlined in this goal.
EXTERNAL EXAMINER INTERVIEWS	Moderate potential. External examiner interviewers usually work from a protocol that should be shaped by the department's interest in the effectiveness of application skills.
FOCUS GROUPS	Strong potential. Focus groups can be used to gather initial data that may zero in on a specific problem. As such, the purpose of the group may be to solve a problem and provide feedback to the department based on the expressed purpose. As such, students can apply principles and concepts in psychology both in the process and product of the focus group.
FOLLOW-UP ALUMNI INTERVIEWS	Limited potential. Calling alumni and identifying examples of successful or not so successful applications of psychology can be a source of data, but the demand characteristics of the situation may produce false positive data. If the purpose is not expressly identified by the researcher, the interview may be suspect on the basis of its potential deception.

ARCHIVAL MEASURES	Mixed potential. In most cases, archival measures cannot provide information about the student's ability to apply psychology. At best, archival records may reveal the intention of course design to address application skills.
TRANSCRIPT ANALYSIS	Limited potential. Transcript analysis might yield the percentage of students engaged in "applied" courses (e.g., internships) as well as the quality of their performance in the class, which could provide a diffuse measure of application skills.
ANALYSIS OF TRANSFER PATTERNS	Not applicable.
SYLLABUS AUDIT	Strong potential. A syllabus audit would be a good first step in determining whether or not faculty are requiring students to engage in assignments that require the application of psychological principles and methods. Where application skills have been identified as a goal by the department, this outcome should be reflected in a reasonable number of syllabi or the department will need to re-examine their curriculum offerings or mission.
DEMOGRAPHIC DATA ANALYSIS	Not applicable.
ALUMNI DATABASE	Limited potential. Determining the percentage of alumni who enter professions that require the application of psychological knowledge and skills would allow a department to get a sense of how successful its curriculum is in preparing students to apply psychology on-the-job. However, the link between job title and application of psychology principles may be fuzzy even for the former student.
LIBRARY STATISTICS USAGE/WEB HITS	Not applicable.

References

Palomba, C. A. & Banta, T. W. (1999). Assessment essentials: Planning, implementing, and improving assessment in higher education. San Francisco: Jossey-Bass Publishers.

CHOOSING ASSESSMENT METHODS

Goal 5: Values in Psychology

Value empirical evidence, tolerate ambiguity, act ethically, and reflect other values that are the underpinnings of psychology as a science.

5.1	Recognize the necessity for ethical behavior in all aspects of the science and practice of psychology.
5.2	Demonstrate reasonable skepticism and intellectual curiosity by asking questions about causes of behavior.
5.3	Seek and evaluate scientific evidence for psychological claims.
5.4	Tolerate ambiguity and realize that psychological explanations are often complex and tentative.
5.5	Recognize and respect human diversity and understand that psychological explanations may vary across populations and contexts.
5.6	Assess and justify their engagement with respect to civic, social, and global responsibilities.
5.7	Understand the limitations of their psychological knowledge and skills.

ASSESSMENT CATEGORY

ESTIMATE OF METHOD'S POTENTIAL

CLASSROOM/COURSE DATA	Not recommended overall. Although the subtypes demonstrate differential opportunities for assessing values, in general, classroom and course data support other goals more effectively. Direct inquiry into values may be vulnerable to demand characteristics. Inferring values from indirect methods may be prone to interpretive error.
OBJECTIVE TESTS	Not recommended. It may be possible to assess values using this technique but it is unlikely to yield an accurate assessment of the student's true commitment to scientific values.
ESSAY TESTS	Limited potential. Questions that are specifically targeted to inferring and discussing relevant science values may be somewhat helpful, but again the demand characteristics may distort validity.
EMBEDDED QUESTIONS AND ASSIGNMENTS	Limited potential. Departments can embed values checkpoints at various points in required courses, but demand characteristics may influence students to respond in socially desirable ways rather than what they truly believe.
CLASSROOM ASSESSMENT TECHNIQUES	Not recommended. Most classroom techniques concentrate on capturing student understanding of content or appraisal of class effectiveness. Their values may be inferred in the latter purpose but those data tend to be of secondary interest in this application.

INDIVIDUAL PROJECTS/PERFORMANCE ASSESSMENT	Mixed potential overall. Individual projects and other forms of performance assessment do provide some opportunity to assess values based on how students develop their arguments and express what they have learned. Errors (e.g., reliance on personal experience vs. empirical evidence) may provide the basis for a strong inference about which scientific values have not been embraced. Faculty may feel uncomfortable offering feedback based on inferences, no matter how compelling.
WRITTEN PRODUCTS	Strong potential. Written work can assess values in psychology either by addressing explicit values as the focus of the writing or by making errors that reveal the notable absence of an expected value (e.g., when students reports that an experiment "proves" a hypothesis. The presence of designated scientific values in writing projects tends to enhance the overall evaluation of quality of the work since the voice of the paper reflects the values of the community.
ORAL PRESENTATIONS	Strong potential. Oral presentations can also provide significant information about the degree to which students adhere to the values of the psychological community either by the direct values espoused in the presentation or the errors that reveal either a misunderstanding or rejection of those values. Typically faculty do not directly grade presentations based on the values expressed; however, speeches and presentation that more accurately reflect psychology values may exert a positive influence on the grade and feedback.
GRAPHIC TESTS AND DISPLAYS	Limited potential. The abstract nature of values does not lend itself as readily to this type of assessment.
POSTERS	Limited potential. Unless the assigned poster addresses values in an explicit way, faculty may have to infer relevant values from posters designed to address other more concrete concepts. In addition, spontaneous discussion about the poster production can probe student values as one source of data about how the students solved the problem. However, this situation lends itself to strong demand characteristics so students may report the values that will make the instructor happy, not necessarily the true values that motivate their behavior.
STRUCTURAL/SITUATIONAL ASSESSMENTS	Strong potential. Structured problem-solving scenarios can be designed around the scientific values specified in this goal. Students can be asked to resolve some value conflicts in ways that will illustrate whether they have integrated the common values psychologists most typically espouse.

SUMMATIVE PERFORMANCE ASSESSMENT	Mixed potential overall. This category contains some methods that show especially good potential for assessing the values that the student has learned to honor, but other methods are less profitable. The department will need to determine their comfort level with whether and how to assess scientific values.
STANDARDIZED TESTS	Not recommended. There are no commonly used tests of psychological values. There may be more generic scientific value inventories but these have not been implemented.
LOCALLY DEVELOPED TESTS	Not recommended. There are no known inventories on psychological values that have been developed in local contexts.
CAPSTONE EXPERIENCES	Strong potential. The design of the capstone experience can and perhaps should include a component in which the student actively identifies the ways in which their work in the course actively expresses scientific values. Values and ethics may be the content base for the capstone as well, which provides several venues to assess what the student recognizes, understands, and practices among scientific ethics.
INTERNSHIP/PROFESSIONAL APPLICATIONS	Strong potential. Settings in which students can make practical applications of the psychology content they have learned can illustrate the degree to which they have accepted and practiced scientific values. However, not all internship supervisors are likely to have a keenly developed sense of the values expectations. Therefore, a survey or set of criteria may be most useful in framing feedback on the degree to which students illustrate those principles.
PORTFOLIOS	Strong potential. One criterion that can drive selection of work for the student portfolio is the degree to which those products illustrate psychological values. The criteria add legitimacy to faculty comments about the salience of the values in student performance. Including a self-assessment dimension may further foster student's understanding of the critical values.
ASSESSMENT CENTER METHODS	Mixed potential. Not all assessment center methods directly address values. However, it may be possible for students to justify the actions they take in performance assessment situations from the values to which they subscribe. For example, the student might be asked to simulate serving as an editor who needs to choose among three articles, which differ dramatically in the degree to which those papers adhere to the prescribed values. The justification for the selection would reveal the student's values.
CASE AND LONGITUDINAL STUDIES	Strong potential. Qualitative studies of student's changes over time could capture shifts in their adherence to scientific values although traditionally the emphasis in such studies tends to be more focused on the value of the content base itself. Departments would benefit from knowing what elements of the curriculum foster improvements in the practice of scientific values. For example, in which courses are students likely to become comfortable with complexity and ambiguity? Well-framed qualitative investigation would yield such answers.

SELF-ASSESSMENT	Mixed potential overall. Many departments do not pay direct attention to the assessment of psychological values since these may be regarded as abstract or esoteric. In such situations, students will be less well prepared to self-assess. However, to the extent that departments can clarify their expectations about the ways in which they expect student values to change toward greater appreciation of the scientific aspects of psychology, the more student self-assessment can be facilitated.
STUDENT JOURNALS	Mixed potential. In student journals, values can be addressed directly or they may be inferred based on student discussion of related phenomena. Better journal entries will be framed in ways that students can directly discuss their practice of identified values.
SELF-CRITIQUES	Strong potential. Where departments make their values & expectations explicit, students should be able to evaluate the extent to which their own work matches the department's expectations.

COLLABORATION	Limited potential overall. The assessment of the degree to which individuals express scientific values may be challenging to assess in group contexts. Even when the focus of the group activity is directly linked to values, discussion about values may not predict individual behavior. On the other hand, conflict situations may clarify the degree to which students differ in the values that they profess.
RESEARCH TEAMS	Limited potential. Students who work under the direction of a research mentor are likely to receive indirect training on the scientific values that undergird high quality research. When students are challenged to explain why certain actions are required as part of the research process, their understanding and adherence to scientific values can be assessed.
GROUP PROJECTS	Limited potential. Most group projects that transpire in the undergraduate curriculum are unlikely to address scientific values directly. However, some projects could be designed that would allow students to solve problems in such a way that their collective grasp of scientific principles could be demonstrated.
ON-LINE ACTIVITIES	Poor potential. Values may be inferred from group process but the amount of work required make this assessment approach untenable.

INTERVIEWS & SURVEYS	Mixed potential overall. Various approaches in this goal produce differential outcomes in identifying values.
SATISFACTION SURVEYS	Limited potential. Satisfaction surveys do not tend to focus on values related to psychology education. Perhaps some survey items could be crafted to address values, but that might detract from the main purpose.
PERFORMANCE REVIEWS	Limited potential. Inferring other's values from their performance is dicey business. Perhaps it is not best to describe definitively what the values related to psychology education might be.
EXIT INTERVIEWS	Strong potential. Although assessing values will be determined by the concentration of the interview protocol, it is possible to have students conduct some targeted reflection on the values that they have embraced during the course of their education. An additional problem is that the values reported during an interview may not be the values of practice.
EXTERNAL EXAMINER INTERVIEWS	Strong potential. Again, the success in identifying values is dependent on the design of the protocol. It is possible to gain some insight about how values have changed, but personal reports may not correspond to performance realities beyond the interview.
FOCUS GROUPS	Strong potential. A focus group can be convened to address how values change as part of education. However, focus groups tend to have a problem-solving focus apart from values.
FOLLOW-UP ALUMNI INTERVIEWS	Limited potential. Discussing values-related issues with alumni may be a window into their values structure, but the complications of this method, including the problem of deception, make it a less .

ARCHIVAL MEASURES	Not recommended overall. Archival measures generally cannot provide a good gauge of values professed or practiced by psychology students.
TRANSCRIPT ANALYSIS	Not recommended. Although it is possible to infer some values from course selections, there are two many variables that influence course choice for the inferences to be meaningful. In addition, adhering to scientific values cannot be assumed just because science courses have been completed.
ANALYSIS OF TRANSFER PATTERNS	Not recommended. Values cannot be inferred from past coursework.
SYLLABUS AUDIT	Not recommended. Most faculty do not explicitly address the values that a course promotes so an audit is unlikely to produce helpful data about values in psychology education.
DEMOGRAPHIC DATA ANALYSIS	Not recommended. Demographic databases are unlikely to address values in a direct and meaningful manner.
ALUMNI DATABASE	Not recommended. Tracking values explicitly is not a typical feature for the alumni database and doing so could be construed as invasive.
LIBRARY USE/WEB HITS STATISTICS	Not recommended. A checked-out library book or evidence of a web hit does not guarantee that the content has been examined or has created any influence.

CHOOSING ASSESSMENT METHODS

Goal 6: Information and Technological Literacy

Demonstrate information competence and the ability to use computers and other technology for many purposes.

6.1	Demonstrate information competence at each stage in the following process: formulating a researchable topic, choosing relevant and evaluating relevant resources, and reading and accurately summarizing scientific literature. that can be supported by database search strategies
6.2	Use appropriate software to produce understandable reports of the psychological literature, methods, and statistical and qualitative analyses in APA or other appropriate style, including graphic representations of data.
6.3	Use information and technology ethically and responsibly.
6.4	Demonstrate basic computer skills, proper etiquette, and security safeguards.

ASSESSMENT CATEGORY

ESTIMATE OF METHOD'S POTENTIAL

CLASSROOM/COURSE DATA	Mixed potential. The only venue in which classroom and course data might reveal information technology expertise would be classes that are heavily mediated. For example, computer labs might be used for on-line testing that would allow some opportunity to gauge student expertise with this method. For the most part, information skills will be better assessed with other strategies that involve activities outside the classroom.
OBJECTIVE TESTS	Limited potential. Knowledge of information and literacy skills could be tested through objective test questions; however, other methods will demonstrate their understanding more directly.
ESSAY TESTS	Poor potential. In class essays would have to focus on student reports of their information retrieval and technological strategies that would produce extremely boring reading.
EMBEDDED QUESTIONS AND ASSIGNMENTS	Strong potential. Departments may dictate specific milestones that should be embedded in required coursework. For example, faculty may specify that exposure to a psychology research database might be embedded in required 101 classes. In smaller contexts, librarians can be enlisted to help conduct information skills training. Later in the curriculum, faculty might identify a courses or set of courses in which they can commit to a particular length of paper with an explicit minimum of high quality scientific sources. The quality of information skills can be inferred from the product; the quality of technological expertise might require more digging or more explicit reporting mechanisms.
CLASSROOM ASSESSMENT TECHNIQUES	Limited potential. This approach may be helpful only in classes specifically focused on the development of information and technology skills.

INDIVIDUAL PROJECTS/PERFORMANCE ASSESSMENT	Mixed potential. Intellectual products provide a stimulus from which faculty can interpret sophistication in information and technological skills. When construction is remote, there may be some risk that the student's work is not an individual production. However, faculty can address this limitation by including a reflective piece that directly addresses the targeted skills.
WRITTEN PRODUCTS	Optimal potential. Written projects provide an ideal context in which to look at research generation, information evaluation, and technology skills.
ORAL PRESENTATIONS	Moderate potential. Individual oral presentations provide some opportunity to evaluate quality of resources; however, these may be challenging to judge in the moment. oral presentations do provide an opportunity to examine power-point or overhead management. In addition, the coherence and development of an oral presentation can reveal research strategies.
GRAPHIC TESTS AND DISPLAYS	Limited potential. Exploring concepts through graphics tends to be an intermediate step in developing research ideas. These may be difficult to assess quality. Such displays may or may not provide an opportunity to assess technology skills,
POSTERS	Moderate potential. The limited space available in most posters may not provide an ideal context in which to evaluate the process of generating research ideas. The poster normally produces highlights so errors or suspect variations may be harder to determine. The execution of the poster will require some technological and aesthetic skills to be successful.
STRUCTURAL/SITUATIONAL ASSESSMENTS	Limited potential. Situational assessments move remote activities into observable territory to facilitate faculty assessment. However, a situational assessment that covers all the outcomes associated with the goal is likely to be fairly intimidating. Performance anxieties may complicate student's ability to perform these complex skills in a situational assessment.

SUMMATIVE PERFORMANCE ASSESSMENT	Mixed potential. Strategies in this category range from zero to maximally helpful.
STANDARDIZED TESTS	Not available. Although there is no standardized approach for measuring research skills, this gap represents an interesting development opportunity.
LOCALLY DEVELOPED TESTS	Limited potential. Some departments have developed methods to assess information skills and research summarizing skills. Assessing these abilities using objective means will be efficient. Asking students to summarize literature will be more challenging and time-consuming but still do-able.
CAPSTONE EXPERIENCES	Strong potential. Assuming that the capstone course has an expressive requirement (e.g., writing or speaking), the capstone course can provide an integrated demonstration opportunity.
INTERNSHIP/PROFESSIONAL APPLICATIONS	Limited potential. The focus of most applied summative experiences will not be focused on the development of the targeted skills.
PORTFOLIOS	Strong potential. Providing explicit criteria that ask students to select �works� based on what these reveal about their skills. The evolution of their abilities can be the focus of reflection on why they selected the items they did.
ASSESSMENT CENTER METHODS	Limited potential. Assessment center methods, like situational assessments, move remote activities into observable territory to facilitate faculty assessment. However, a situational assessment that covers all the outcomes associated with the goal is likely to be fairly intimidating. Performance anxieties may complicate student's ability to perform these complex skills.
CASE AND LONGITUDINAL STUDIES	Limited potential. The complexities of tracking these skills over time mitigates against adopting this strategy to monitor their evolution.

SELF-ASSESSMENT	Mixed potential. Developing student's metacognition of their achievement in information and technology skills has not been overtly addressed through self-assessment strategies.
STUDENT JOURNALS	Limited potential. It is unlikely that faculty would choose to invest time reading about student's struggle to learn information and technology skills. However, engaging in student journal writing might provide some keys to faculty about where the particular points of struggle might be.