Using the Global Assessment of Functioning Scale to Demonstrate the Importance of Inter-rater Reliability

Principle Demonstrated

Inter-rater reliability is a measure of consistency used to evaluate the extent to which different judges agree in their assessment decisions. Inter-rater reliability is essential when making decisions in research and clinical settings. If inter-rater reliability is weak, it can have detrimental effects.     

Purpose

Inter-rater reliability is an important but often difficult concept for students to grasp. The aim of this activity is to demonstrate inter-rater reliability. Furthermore, students learn why it is important and how to increase it. I designed this activity for my Introduction to Psychology and Abnormal Psychology courses.

Equipment

  • The Global Assessment of Functioning Scale

  • Background and Demographics Information of Fictitious Patient

  • Blackboard

  • Chalk

Preparation

I created a description of a fictitious patient, Cameron. My 500-word description includes demographic information as well as details about Cameron’s interpersonal relationships and professional endeavors. My description is included with my submission.   

Procedure

After defining inter-rater reliability on the board, I inform the students that I have just “hired” them to help me screen potential patients. I explain that as employees they will need to rate each potential patient using the Global Assessment of Functioning Scale. Their findings will determine what kind of care a patient receives: individual therapy, partial hospitalization, inpatient hospitalization or no care at all.     

I tell them that their first assignment is to rate Cameron using the scale. A copy of the scale is provided to students. I intentionally provide a cursory description of how the scale is used; untrained raters will enhance the results given that the activity’s purpose is to demonstrate variability of scores. After reading the description of Cameron, students must individually select a score based on their perception of his functioning. They are not allowed to discuss their thought process with the other students at this point in the activity. 

Students are given five minutes to complete their task. Once the time has allotted, I ask each student for their score and write it on the board. Typically their answers will vary at least 50 points.

Discussion is then elicited. Students are asked to explain their rationale for selecting the score that they did. Particular focus is paid to students who gave the highest and lowest scores; these two students have a very different view of the same patient. 

I Then Ask the Following Questions and Elicit the Following Answers From the Students:

Did you achieve inter-rater reliability?

No, there was too much variability in scores.  

What are the implications of poor inter-rater reliability?

The care a patient receives is dependent on who judged them and not on their actual functioning. One rater might have recommended inpatient hospitalization whereas another might have seen psychotherapy as unnecessary.      

How could I have achieved better inter-rater reliability?

Training the employees on how to use the Global Assessment of Functioning Scale could have enhanced reliability.

Once inter-rater reliability is achieved is it maintained over the course of time?

No, rater drift can occur. Rater drift is when the raters return to their previous tendency of rating. 

How can we prevent rater drift and ensure inter-rater reliability?

By providing frequent trainings over the course of time, we can maintain inter-rater reliability.            

Ethical Problems

I have encountered no ethical problems with this activity.

Background and Demographics Info of Fictitious Patient

Cameron is a 30-year old Caucasian man who lives with his girlfriend in a middle class community.  Since having moved out of his parents’ home, he speaks with them infrequently. His mother is a homemaker and his adopted father is an engineer. Cameron has been estranged from his biological father and his paternal grandparents since he was 8 years old when his mother remarried. 

Cameron’s biological parents divorced when he was one year old. Consequently, his biological father moved to Florida. Cameron and his mother moved into his maternal grandparents’ house on Long Island. The geographic distance prevented Cameron from seeing his biological father more than once or twice a year. Cameron stated, “I was never really close to my father, so it wasn’t a big deal when I never saw him again.” When his mother remarried, the man adopted Cameron and they moved into their own home. Cameron is unclear to whether it was the adoption which caused his father to cease contact with him or some other event. During his childhood, Cameron describes his parents as “asleep at the wheel.” According to Cameron, they were unaware of his marijuana use during high school and never expressed concern over the friends with which he associated.

Cameron is currently working as a divorce lawyer. Always a stellar student, Cameron has received numerous accolades throughout his education. Currently, he is employed by a prestigious law firm. When asked about his job Cameron stated, “I’m under enormous pressure. I frequently work over 60 hours a week. I don’t want it to be, but my work is my life.” He also expressed that he finds himself to be “confrontational and combative” with others when under stress at work. In fact, he reports that the last three secretaries have quit because of his behavior. Cameron also has a strong desire to build a private practice, but fears failure. 

In order to relieve stress, Cameron plays guitar. About once every three months, he and his band play at various venues in New York City. At these gigs, he often “lets loose” by doing many shots of vodka. He and his girlfriend frequently argue over this use of alcohol. She feels that he is acting irresponsibly and immaturely. He feels that she just doesn’t understand the level of stress that he is under. Otherwise, Cameron reports that his relationship with her is positive. 

Cameron is currently seeking an evaluation to determine whether he is a candidate for services and if so what type.