Also in this Issue

Evaluating the U.S. Naturalization Test Redesign

The Committee on the U.S. Naturalization Test Redesign, convened under the auspices of BOTA, will provide guidance to USCIS to help optimize the validity, reliability, and fairness of the redesigned tests of English and U.S. history and government.

By Monica Ulewicz

How many stripes are there in the U.S. flag? What is the Constitution? What is the most important right granted to U.S. citizens? Under the current naturalization test process, applicants for U.S. citizenship are required to answer questions like these to demonstrate an understanding of the fundamentals of U.S. history and government. Applicants must also read, write, speak and understand words in ordinary usage in the English language. Applicants are tested on these requirements during an interview process with U.S. Citizenship and Immigration Service (USCIS) officers, but the content and process can vary from officer to officer and from district to district. Given concerns that the current testing process may not be sufficiently uniform and that content may not be meaningful enough, USCIS is currently redesigning the naturalization test. The naturalization tests affect on average 800,000 applicants per year and any changes will be controversial. USCIS recognized the need for the redesigned process to be based on the highest professional testing standards and approached the Board on Testing and Assessment (BOTA) at the National Academies for independent advice.

The Committee on the U.S. Naturalization Test Redesign, convened under the auspices of BOTA, will provide guidance to USCIS to help optimize the validity, reliability, and fairness of the redesigned tests of English and U.S. history and government. The committee will also provide an overall assessment of that validity, reliability, and fairness before the nationwide implementation of the tests tentatively scheduled for 2006. The committee's conclusions and recommendations will take into account the practical, financial, legal and political constraints that limit both the types of tests that USCIS can propose and the length and cost of the process for developing those tests. Committee members represent a breadth of related expertise in testing, American history, political science, immigration policy, language assessment and adult education. The committee is co-chaired by Lorraine McDonnell, Professor of Political Science at the University of California at Santa Barbara, and Barbara Plake, Professor and Director of the Buros Center for Testing at the University of Nebraska. Among the testing experts on the committee, Barbara Plake and Paul Sackett (University of Minnesota) are former members of APA's Committee on Psychological Tests and Assessment.

The Committee on the U.S. Naturalization Test Redesign provided initial formative guidance in an interim report released to the public in December 2004. Redesigning the U.S. Naturalization Tests: Interim Report addressed concerns about the lack of both a necessary advisory structure for making decisions and a coherent, overarching research and test development design that will lead to a scientifically and politically defensible testing program. Among the committee's recommendations was the establishment of an advisory structure, including: a policy or oversight committee to provide political legitimacy and policy directions; a technical panel to provide psychometric and test design advice; and content panels to guide development of the content framework and test specifications and to review test items.

In its report, the committee referenced the Standards for Educational and Psychological Testing (1999) to address its concerns about the naturalization test redesign process. In the committee's judgment, the redesign program has lacked a clear statement of the purpose of the tests, with no clear operationalization of the constructs embodied in the legislative requirements and no systematic specification of the inferences about naturalization applicants to be drawn from the results. The committee also voiced concern about planned activities, including a Phase 2 pilot study in which pass/fail decisions about applicants would be made based on their performance on a set of items being tried out for the first time. The committee recommended that the redesign program should be guided by a coherent research and test development plan that complies with testing standards and includes all the necessary steps for developing a valid, reliable, and fair test. In addition, the committee recommended a clear, transparent and publicly accountable process to develop the content frameworks given the vague and contentious nature of the legislative language defining the constructs, i.e., "reading and writing simple words and phrases" or "understanding the fundamentals of history."

The committee is currently in its data gathering phase. It held a workshop in December 2004 to hear a broad range of perspectives about the redesign and conditions in the field that will likely affect its implementation. Perspectives included community-based organizations, immigrant advocacy groups, immigration lawyers, USCIS officers, state and local government agencies, and national policy groups. The committee has also invited input from any interested organizations in the form of brief, two-page statements. The committee's fact-finding, data-gathering phase will likely continue for another two or three meetings during 2005, with a final report tentatively scheduled for release in 2006. More information about the committee, including its charge, membership, and activities can be found on the National Academies Board on Testing and Assessment web site.
Submitted by National Research Council staff member Monica Ulewicz with input from the committee co-chairs, Barbara Plake and Lorraine McDonnell.