Advantages:
+ facilitates student command of specific area
+ provides practice in critical skill area of writing
Disadvantages:
- challenging to writing-compromised students
- labor-intensive to score and return with timely feedback
- can be plagiarized created time-consuming/strategic confrontation with serious consequences for students who are caught
- instructors can be plagued with consequences of student procrastination
Recommendations:
Many professors design writing projects in stages that promote multiple drafts. Getting feedback in stages may be easier for students to incorporate and easier for faculty to see the impact of their feedback work. Learning disabled, ESL, and other writing challenged students may require additional support. Efficient feedback can be facilitated using rubrics or style sheets. Writing projects should be tailored to the developmental level of the student. For example, beginning courses can employ letters to friends to explain a concept. Formal term papers typically work best in advanced courses. Departments may adopt a style sheet based on APA writing conventions that can help students practice consistent format strategies.
ORAL PRESENTATIONS
e.g., debate, role play
Advantages:
+ builds expertise in important communication area of oral expression
+ promotes importance of sharing knowledge
+ enhances oral skills
+ Q & A promotes Òthinking on your
feet
+ assists professor to cover course content
Disadvantages:
-may burden students with ESL, speech and language difficulties, speaking
anxiety
-time consuming and time-wasting when work quality is bad or boring
-may be hard to grade
Recommendations:
Students understandably resist assignments that require them to speak in classes since public speaking remains one of our most pervasive social phobias. Success in oral presentations will depend on several elements:
=>providing lots of guidance and structure beforehand
=>normalizing speaking discomfort and pointing out that overcoming those fears can happen only through practice
=>specifying and sticking to assigned time limits
=>circumscribing topic areas or requiring topic approval
=>coaching regarding use of support technologies
=>developing appropriate performance criteria
GRAPHIC TEST AND DISPLAYS
(e.g., concept maps, outlines)
Advantages:
+ provides experience in applying and organizing course concepts
+ assists in thinking through organization of information
+ additional grappling with the material enhances recall
+ appeals to visual learners
Disadvantages:
- students have limited practice with displaying graphic skills
- students may not have sufficient experience in interpreting graphics
- technological sophistication will influence production quality
- may waste resources
Recommendations:
Faculty have found some success in asking students to translate lecture input into graphic displays, such as a concept map. These strategies appeal to visual learners who may be able to encode and remember more course content by adopting this strategy.
POSTERS
Advantages:
+ hold students accountable for independent project
+ reduces grading burden compared to writing projects
+ provides opportunity to integrate communication skills (e.g., writing,
graphics, oral defense)
+ can incorporate team effort
+ expert judgment, peer review can be facilitated with criteria
+ simulates typical debut venue for most psychology scholars
Disadvantages:
-may need to make special arrangements for space
- students may invest money in project for one-shot exposure
- lack of aesthetic sense may handicap poster effectiveness
- stronger social interaction skills may produce halo effect in judging quality
- numbers of posters to be judged can create quality pressures on grading
- may not motivate best effort
Recommendations:
Providing models or performance criteria will facilitate better productions. Poster sessions can be scheduled within classes or across classes as a departmental event. Awarding best of show maybe a helpful strategy to enhance motivation among the best students. All-department events can become a
public relations resource as well as an opportunity to work with local high
school psychology teachers to recruit future students.
STRUCTURAL/SITUATIONAL ASSESSMENTS
(e.g., guided learning, in-baskets, critical situations, etc.)
Advantages:
+ provides realistic testing circumstance
+ reality engages and motivates students
+ promotes transfer of information, application
+ taps complex skills
Disadvantages:
- difficult to construct and measure
- locating designed instruments is challenging
- prone to history/context/age cohort effects
- students may rely on common sense under pressure rather than their
knowledge from the course
Recommendations:
The situation should correspond closely to the learning conditions to promote the best transfer of knowledge. Evaluating performance will be facilitated by clearly developed criteria. The quality of the rubric and the training of the evaluators will influence validity. If inter-rater reliability is not high, the results will be of limited value. Rubrics will sometimes not provide for unexpected, creative responses.
|
SUMMATIVE PERFORMANCE ASSESSMENT
|
NATURE OF CATEGORY:
Summative assessment strategies tend to be employed for purposes of evaluating program quality rather than primarily to provide developmental feedback to
students. This collection of assessment strategies include methods that involve a single episode of data collection (e.g., nationally or locally normed tests) as well as those that incorporate tracking student performance over time (e.g., portfolio, case studies, longitudinal studies. Capstone courses and internships can also be appropriate contexts for summative evaluation.
OVERALL ANALYSIS:
Advantages:
+ promotes coherence in curriculum planning
+ provides feedback loop to improve quality
+ some strategies can be adapted to student interests
+ supports to earlier curriculum recommendations (e.g., St. Mary's conference to
provide vehicle for integrating learning)
Disadvantages:
- some options are labor and/or cost intensive
- students may not receive direct feedback regarding their performances, thus limiting their own gains from effort expended
- departments may ignore available data in their planning
Recommendations:
Summative procedures can be invaluable in making the case for the overall quality of programs. Although all of the methods have advantages and drawbacks, the most benefit can be gained to all constituents when students receive direct feedback regarding their summative performance. Finding out relative
scores on comprehensive exams or receiving feedback regarding performance over
time can assist students with career and life planning in some instances.
STANDARDIZED TESTS
Advantages
+ typically one shot assessment
+ facilitates comparisons over time
+ convenient
Disadvantages
- may not reflect gains or growth across time
- exiting students may not benefit from feedback
- existing instruments may not match to the mission and goals of departments
- expensive
- students may not be motivated to due their best work
- when test occurs may not maximize true learning
- administration may not be flexible
- not student-centered
- limited faculty ownership
- verifying bad performance can be threatening to motivation
- scores may be delayed in return, reducing the impact of feedback
- there may not be a standardized test for the identified content
- can facilitate problematic comparisons to other programs (e.g., comparisons may not take into account differential resources, student characteristics, etc.)
Recommendations:
The disadvantages of the use of standardized tests can be minimized with some
additional planning. Embedding the capstone test in an existing course will enhance student motivation since the student may take the experience more seriously. When student performance can also be tied to course grading, maximum motivation to do well is likely. Describing how well the existing test matched the required curriculum will encourage faculty support and student cooperation.
LOCALLY DEVELOPED EXAMS
Advantages:
+ can be tailored to match curricular and program goals
+ standardizes local use
+ relatively inexpensive
+ provides opportunity to develop meaningful local norms
+ avoids specious comparison with other colleges
+ foster coherence in department about their objectives
+ speedy feedback
+ cheaper than national products
+ after initial investment, saves time in the long run
+ may be embedded in specific standard courses
Disadvantages:
- complex, time-consuming to develop
- may impede curricular change since test would need retooling after reforms
- reliance on test bank may not inadequate due to test bank quality
- vulnerable to student theft and distribution
- can be misused by comparing faculty member's areas
Recommendations:
Comprehensive local exams are very time-intensive on the front end; however, the pay-off for this activity is multiple. This strategy encourages strong collaboration across department members and will help department members learn about the academic goals of their colleagues. Security will be an important issue to keep the department test safe from test files that may exist across campus.
CAPSTONE EXPERIENCES
Advantages:
+ fosters aura of importance that may motivate students throughout the curriculum
+ encourages departmental endorsement of culminating
experience
+ promotes student responsibility for engaged course
+ supports program coherence for faculty and students
+ course content can be flexible
+ topical design of capstone can engage faculty in planning (e.g., seminar topics can be taught in special interest areas as long as the performance goals meet department expectations)
Disadvantages:
- high stakes performance can be impaired by performance anxiety
- typically low enrollment course is expensive to provide seats for all seniors
- faculty can generate territorial concerns over right to teach capstone
- graduation may depend on successful completion of capstone which can generate some anxiety for faculty and students when performance wobbles late in the course
Recommendations:
Departments can use capstone courses as a unique way to express special interests of the faculty. Departments should secure the support of administration for this expensive option before broad implementation. Typically, capstones tend to have small enrollments to maximize faculty-student interaction. Capstones provide a great opportunity to have the student reflect meaningfully over the course of the curriculum. Putting in place some checkpoints on the process may prevent last-minute difficulties in the capstone that can compromise graduation plans.
INTERNSHIPS/PROFESSIONAL APPLICATIONS
Advantages:
+ popular choice for students
+ provides opportunity to sample future career
+ positive public relations vehicle related to well-prepared students
Disadvantages:
- time intensive for faculty mentors to connect with on-site mentors and
coordinate opportunities
- challenging to foster learning experiences across multiple sites
- poorly prepared students create public relations problems
Recommendations:
Departments may reduce the pubic relations strain by screening students for their readiness to represent the program in public contexts. Qualifying criteria that stress quality and quantity of course experience as well as professional expectations in the intern role can set a positive, appropriate tone. Maintaining close contact with on-site mentors can also reduce
unsuccessful student performance.
PORTFOLIOS
Advantages:
+ shows sophistication in student performance
+ illustrates longitudinal trends
+ highlight student strengths
+ identify student weaknesses for remediation, if timed properly
Disadvantages:
- collection will be no better than the quality of collected instruments
- time consuming and challenging to evaluate
- space and ownership challenges making evaluation difficult
- content will vary widely with students
- students fail to remember to collect items
- transfer students may not be in position to provide complete portfolio
- time intensive to convert to meaningful data
Recommendations:
Clear expectations about the purpose and collection responsibilities will help students succeed in using the portfolio method. The works that student select will be more satisfying if the students can compare to established criteria. If the faculty want student portfolios to represent student development over time, they will need to be scrupulous about setting forth the performance demand of collecting and examining works throughout the student's career. The success of the portfolio may be enhanced when students reflect on how all the pieces work together to express their learning or meet department criteria.
ASSESSMENT CENTER METHODS
e.g, in-baskets, guided problem-solving
Advantages:
+ complex tasks can enhance student motivation
+ designing relevant authentic assessment practices challenging
+ facilitates integration of diverse skills and content areas
Disadvantages:
- expensive in material preparation and time
- students may not always perceive relevance of assessment to their studies
Recommendation:
Not all disciplines may lend themselves as readily to problem solving situations that seem to be at the center of those challenges.
CASE AND LONGITUDINAL STUDIES
Advantages:
+ can provide rich detail
+ level of attention can build esteem
+ builds allegiance
Disadvantages:
- transfer students may be omitted
- expensive and time-consuming
- small sample limits generalization
- attribution of historical or cohort effects may taint participant reports
- selection for tracking may influence outcome and change student experience
Recommendations:
Departments need to clarify selection criteria if only a sample of students will be tracked. The results simply may not be representative of the group as a whole. Special care must be taken to have a satisfying instrument if results will be compared across cohorts. A department member may need to coordinate this activity if the department commits to this strategy.
Advantages:
+ multiple modes and variable sophistication possible
+ quality of self-assessment related to quality of content knowledge
+ flexible in format; prompts provided or not
+ might ask about change over time
+ empowers students to practice self-evaluation
+ promotes transfer of accountability to other situations
Disadvantages:
- student judgment may not be accurate
- self-assessment are prone to evaluative biases (e.g., Lake Woebegone Effect,
underestimation due to self-esteem issues)
-students have limited experience being held accountable to judge their own work
- students may define assessment as job of teacher
- faculty may perceive this practice to set up more grade conflicts
Recommendations:
Students should receive feedback on the accuracy of their self-evaluations. Early assignments might fare best with more global criteria. For example,
"what aspects of your performance were effective?" and "What would you do
differently if you had more time?" may engage the student in being reflective.
Over time, students should be able to apply more discrete criteria to their own performance, and eventually they should be able to help formulate criteria by which performances should be judge. The quality of self-assessment may be very dependent on the careful construction of the self-assessment prompts.
RESEARCH TEAMS & GROUP PROJECTS
Advantages:
+ student-centered designs promote engagement
+ provides opportunity to practice group skills, time management
+ promotes independent work at deeper level
+ breadth of assignments can address content coverage issue
+ simulates how professional activities/achievement transpires
+ produces synergy and excitement around project completion
+ creates a venue to synthesize content bases from multiple courses
Disadvantages:
- students have limited training in group dynamics
- social loafers can tax equitable judgments about grading
- erroneous ideas that are not caught and corrected spread across group
members
- challenging to faculty to judge when to redirect or rescue student
groups in trouble
- time-consuming
Recommendations:
Selection of the group members will influence group outcomes. For example, some projects will work best when the groups are heterogeneous with regard to student characteristics. Other projects might be most efficient when groups are homogeneous. Students may need assistance in understanding how groups work. Their work will improve with some prompts to pay attention to the process of the group in addition to solving the problem at hand or creating the product. Students will fare best in research teams where they clearly understand group norms and expectations. For example, what are the penalties for nonparticipation? Whenever possible, students should be given feedback on the quality of their participation.
ONLINE ACTIVITIES
(e.g., maintaining print record of interactions in chat room or other internet-based contact)
Advantages:
+ the data already exist as part of regular course
+ records trends in collaborative skill
+ tracks process
+ cheap and convenient
+ demand characteristics may be reduced
+ students have equal opportunity to participate
+ faculty monitoring can be unobtrusive
+ appeals to some students who may have greater difficulty in oral expression
+ provides archive through automatic recording
+ documents feedback for instructor on what has been covered or what is still unclear
Disadvantage:
- content analysis is time-consuming
- privacy issues can be compromised
- students may be handicapped by computer savvy and tech patterns
- faculty need to be computer savvy
Recommendations:
Instructors using online strategies may need to overcome individual differences in using this mode by requiring participation. Circumscribing the content may help to avoid some ethical challenges that result in chat room participation. Students should be informed that their discussions are being monitored for assessment purposes from the outset. This strategy may entail IRB review to confer the best protection. Faculty also need to assess ease of web
access for students before making on-line participation a requirement.
ATTITUDE MEASUREMENT (General Analysis)
Advantages:
+ easy to administer
+ cheap
+ easy to score
+ quick feedback
+ can be reliable but not valid
Disadvantages:
- validity hinges on good design
- may not be valid
- demand characteristics may distort results
- participants may not have good knowledge about their own attitudes
- participants may demonstrate response bias or dishonesty
- labor intensive to interpret
Recommendations:
Valid attitude measures depend on quality of design and implementation. For example, the participants must be motivated, careful, and candid to generate data that will be meaningful. Care should be exercised to produce to
design appropriate measures for intended purposes that minimize sources of
error (e.g., selection bias, demand characteristics, literacy challenges,
etc.).
SATISFACTION SURVEYS
(Alumni, Employers, Grad School Advisors, Parents, etc.)
Advantages:
+fosters positive public relations because activity signals faculty concern for quality
+targets of survey may be prompted to other positive actions (e.g, donations, hiring, recruitment of new students)
+external judges may be more objective in their appraisal of student abilities, achievements
+recurring insights may point to some problems that need remediation
+provides important perspective on relevance of program to various occupations
Disadvantage
- tracking down and engaging targets may be problematic
- low return rates compromise validity
- some respondents may be motivated not to tell the truth (e.g., don't want to bear bad news, demand characteristics)
Recommendations
Long surveys will influence completion rate. The return rate also provides some indication of how robust the results are. For example, in alumni surveys,
the students who are most successful will be more motivated to complete the
surveys and may produce an overestimate. When appropriate, a lie scale or some other strategy to verify truthfulness in response will also increase validity.
In designing satisfaction instruments, instructors need to think through the quality of education from the perspective of the interview subject. Well-designed surveys are difficult to create so some pilot data may help identify trouble spots in proposed instruments.
PERFORMANCE REVIEWS
(Alumni, Employers, Grad School Advisors)
Advantages:
+ promotes evaluation based on objective appraisal of behavior
+ builds positive public relations
+ external judges may be more objective in their appraisal of student abilities, achievements
+recurring insights may point to some problems that need remediation
+provides important perspective on relevance of program to various occupations
Disadvantage
- tracking down and engaging targets may be problematic
- low return rates compromise validity
-some respondents may be motivated not to tell the truth (e.g., don't want to
bear bad news, demand characteristics)
Recommendations:
Departments committed to evaluating their graduate's performance from interested
stakeholders are likely to find the time invested to be worthwhile, both in
terms of data gathered as well as public relations impact.
EXIT INTERVIEWS
Advantages:
+ provides realistic picture
+ provides catharsis
+ provides in-depth, personal perspective on experience of major
+ can be embedded in existing courses to capture broad range of student experience
+ demonstrates overt department commitment to high quality
+ may promote long-term allegiance among graduating students
+ can generate reinforcing feedback to help departments sustain effectiveness
Disadvantages:
-volunteers may have a negative or a positive agenda that may not be
representative, producing a selection bias
-time-consuming to coordinate and evaluate the results
-students may not show up for discussion
- negative discussion may influence formerly neutral students to redefine their experience negatively
- completion challenge
- participants may paint too rosy a picture partially due to timing
- expensive
- results can be influenced by the quality of the interviewer and protocol
Recommendations:
Departments will need to decide on the scale and format of focus exit interviews. These activities can be conducted individually or in small groups.
Departments can commit to interviewing every graduating seniors or elect
to sample from the group. Instructors need to determine how much credence to place on the results of group discussions with students based on sample size and representation. Questions should target the data that the department wishes to gather. The department should also determine how to interpret the results of the
interview. Collaborative design of the interview protocol will promote greater enthusiasm by department members to deal with the consequences of the interview. Conducting the interviews with department faculty may influence student participation since they may be more candid with an external reviewer.
FOCUS GROUPS
Advantages:
+small discussion groups promote engagement
+ can be employed to provide feedback on a class, course, or program
+ participants can benefit directly from changes that result from their feedback
+ demonstrates overt department commitment to high quality
+ can generate reinforcing feedback to help departments sustain effectiveness
+ development of protocol can be involving for faculty
+ may tap unforeseen areas of concern
Disadvantages:
- current students may feel some pressure not be completely candid for
fear of retribution
-volunteers may have a negative or a positive agenda that may not be
representative
- time-consuming to coordinate and evaluate the results
- students may not show up for discussion
Recommendations:
Departments should develop a good rationale for selecting students for focus group linked to the purpose for which the group is being convened. The discussion protocol can produce both quantitative and qualitative data that can be beneficial to the department. However, student commentary in a focus
group may not be representative of the typical student's experience.
FOLLOW-UP ALUMNI INTERVIEWS
(This method involves telephone follow-up to graduates to assess information other than satisfaction with the major. Graduates can be contacted and interviewed on various outcome measures, including knowledge of the major, civic practices, or other indices of interest to the department. Demand characteristics are strong in this strategy.)
Advantages:
+ facilitates spontaneous assessment of studentÕs application of knowledge & skill
+ measures enduring learning and skill transfer
+ scope can be broad-ranging
Disadvantages:
- could be construed as deceptive practice
- might require IRB oversight
Recommendations:
Avoiding demand characteristics is a significant problem with this approach. Alumni may feel compelled to help out by inflating their accomplishments or satisfactions in response to a phone interview.
EXTERNAL EXAMINERS
(Exit interviews conducted by objective, external expert)
Advantages:
+ promotes objective reports where students are assured of anonymity
+ data summary and interpretation conducted external to regular department
activities
+ improves face validity of assessment activities
+ supports department courage regarding willingness to expose their
practices to outsider
Disadvantages:
- expensive to employ qualified consultant
- sensitive information is at some risk for getting beyond control of department
Recommendations:
Departments may want to involve the external examiner in the construction of the interview protocol to avoid problems of drift toward the examiner's own interests and values in the interview. Qualified external examiners can be identified through the Psychology Department Consulting Bureau operated by the Society for the Teaching of Psychology.
TRANSCRIPT ANALYSIS/ANALYSIS OF TRANSFER PATTERNS
(can answer questions about prerequisites, transfer patterns)
Advantages:
+ existing data
+ provides overall picture
+ trends of targeted students at particular times
+ exposes problematic trends for transfer, including drop out rates, time to degree completion, course articulation success, subsequent courses performance
Disadvantages:
- time-consuming
- potentially boring in level of detail required
- may require cooperation to gain access to data
Recommendation:
The analysis of course patterns by itself may not address directly the questions regarding quality. Transcript analysis can answer narrowly focused questions that should be well thought through to justify the time required.
SYLLABUS AUDIT
Advantages:
+ promotes coherence within the department
+ can identify areas of neglect or overemphasis
+ facilitates adoption of similar writing standards and other expectations
+ promotes student understanding of cognitive goals
Disadvantages:
- time-consuming
- may be difficult to engage all department members fully in review/consensus
- students may pay little attention to the syllabus as overall learning guide
Recommendations:
Although this practice is time-consuming, many departments find a syllabus audit is fundamental to answering all kinds of questions about the manner in which the
faculty implement the curriculum.
DEMOGRAPHIC DATA ANALYSIS/ALUMNI DATABASE
Advantages:
+ facilitates thorough understanding of student body
+ prepares department for unusual trends that might affect course scheduling
+ predicts where recruitment efforts will pay off
+ points to specific remediation needs
+ identifies potential donors for ongoing program needs
Disadvantages:
- time-consuming
- possible too have too much data
Recommendation:
With careful planning, departments can execute well crafted strategies to collect data that will be useful for their planning in recruitment, retention, and fund-raising.
LIBRARY USE/WEB HITS
Advantages:
+ provides input about how seriously students take assignments
+ allows analysis of trends in use
+ presents overall picture of value
Disadvantages:
- contaminated with faculty use
- interpretation is difficult, boring, and time-consuming
- students may get sources from other than the library
Recommendations
This measure may be most helpful feedback from the library to assist in future
ordering. Combining library use and web hit statistics with other measures may provide more meaningful measure.
Return to Assessment CyberGuide home page
