Fairness a challenge when developing special-needs tests

Psychologists search for ways to assess the skills of special-needs students, yet preserve a test's validity.

By Beth Azar
Monitor staff

When special-needs students are granted more time to take the SAT, does that give them an unfair advantage over other students? Or does it merely level the playing field, allowing the students' true abilities to shine through their disabilities?

That's the dilemma facing test developers, many of them psychologists, as they try to create valid, meaningful tests and at the same time accommodate people with disabilities.

Many researchers are looking for solutions to that problem in response to the 1997 revision of the Individuals with Disabilities Education Act (IDEA), which requires states and districts to make educational tests available to students with disabilities by 2003. Meanwhile, testing administrators are also seeking to design fair tests for the increasing number of test takers who are not native English speakers.

Developing accommodations for these students is no easy task, say psychologists who research and develop tests. Changing a test, whether by extending the time someone has to take it, allowing people to read items aloud or providing translations or glossaries, may change its validity. And, testing experts say, an altered exam may no longer assess the same ability or skill it was designed to measure or the outcome it's designed to predict, testing experts say.

Until IDEA forced the issue, researchers gave little attention to how accommodations may influence a test's validity. Now, with grant money emerging from the U.S. Department of Education's

Department of Special Education and Office of Educational Research and Improvement, an increasing number of researchers are examining the problem. Testing administrators estimate that they will need to design accommodations for up to 15 percent of the nation's children--11 percent for students with disabilities and 4 percent for those with limited English proficiency, according to the latest statistics from the U.S. Department of Education in 1996.

Some of these researchers can already point to accommodations that appear to preserve a test's validity while allowing people with special needs to demonstrate their true abilities.

"The research base is growing, with new researchers entering the field all the time," says psychologist and testing accommodations researcher Gerald Tindal, PhD, of the University of Oregon. "And we're starting to figure out what works best."

A differential advantage

One of the most challenging issues for testing researchers has been how to evaluate whether a testing accommodation worked. Most agree with testing researcher Susan Phillips, PhD, of Michigan State University, who argues that an accommodation must provide a "differential advantage"--that is, it must boost the scores of students with special needs above and beyond the gains that students without special needs might obtain. For example, giving extra time for test-taking likely helps all students improve their scores. But the question is, does extra time improve the scores of learning disabled students even more because it helps them overcome their disability and show their true abilities?

Another way to evaluate an accommodation is to examine its effect on a test's ability to predict the construct it was designed to predict. For example, several studies have looked at whether accommodations affect the SAT's ability to predict college performance. In those studies, the most common accommodation has been allowing students with special needs extra time for test-taking. But data on this strategy's effectiveness are mixed.

Lynn Fuchs, PhD, and her colleagues at Vanderbilt University, for example, find that giving more time for conventional math and reading tests doesn't help grade-school students who have learning disabilities any more than it does nondisabled students. But it may provide a differential advantage to learning-disabled students when they take more complicated math tests that require extensive reading and writing, they find.

Meanwhile, studies conducted by the College Board find that providing more time on the SAT allows learning-disabled students to improve scores by 45 points on verbal and 38 points on math. But the findings are inconclusive because no study has compared these higher scores to how students without disabilities might fare with more time, says Wayne Camara, PhD.

"It may be that extended time provides students with disabilities an unfair advantage or it may be that the extended time compensates for the disability and permits students to perform better," says Camara, lead author on the study and executive director for research and development at the College Board.

Researchers already suspect that providing extra time damages the SAT's ability to predict college performance, says Ellen Manindach, PhD, of the Educational Testing Service, who worked with Camara on the College Board study. Several studies, she says, found that the longer students had to take the test, the more their scores overpredict college success.

In the end, students with disabilities will likely require accommodations tailored to their specific needs, says Vanderbilt's Fuchs. Although she and other researchers have identified accommodations that appear to preserve test validity, they're also finding that the same accommodations work appropriately for some students but not others.

For example, providing extended time does not on average boost the scores of students with learning disabilities more than the scores of students without disabilities, researchers find. Yet the accommodation does in fact benefit some individual students with learning disabilities, suggesting that extended time speaks to something essential about the specific disabilities of some subset of learning disabled students.

On a related note, Fuchs and Tindal find that fourth graders with learning disabilities who take math tests benefit more than students without disabilities when someone reads the text on those tests aloud to them. But this accommodation doesn't work as well for seventh-grade students.

"We're not going to find accommodations that work across the board for all students at all ages," says Tindal.

When words get in the way

Students who are learning English as a second language often have trouble demonstrating their true abilities in math and science on standardized tests because test language inhibits their performance. The goal of accommodations for these students is to design a test that evaluates their cognitive abilities without penalizing them for their problems understanding English, says Jamel Abedi, PhD, of the University of California, Los Angeles Center for Research on Evaluation, Standards and Student Testing (CRESST).

He and his colleagues have conducted a series of studies examining accommodations for English language learners on questions borrowed from the National Assessment of Educational Progress (NAEP) mathematics assessment, administered annually by the U.S. Department of Education. The researchers compared the scores of native English speakers to scores of students learning English as a second language, who were allowed one of four accommodations: