Addressing equity and ethics in artificial intelligence

As artificial intelligence (AI) rapidly permeates our world, researchers and policymakers are scrambling to stay one step ahead. What are the potential harms of these new tools—and how can they be avoided?

“With any new technology, we always need to be thinking about what’s coming next. But AI is moving so fast that it’s difficult to grasp how significantly it’s going to change things,” said David Luxton, PhD, a clinical psychologist and an affiliate professor at the University of Washington’s School of Medicine who spoke at the 2024 Consumer Electronics Show (CES) on Harnessing the Power of AI Ethicallyopens in new window.

Luxton and his colleagues dubbed recent AI advances “super-disruptive technology” because of their potential to profoundly alter society in unexpected ways. In addition to concerns about job displacement and manipulation, AI tools can cause unintended harm to individuals, relationships, and groups. Biased algorithms can promote discrimination or other forms of inaccurate decision-making that can cause systematic and potentially harmful errors; unequal access to AI can exacerbate inequality (Proceedings of the Stanford Existential Risk Conference 2023, 60–74opens in new window). On the flip side, AI may also hold the potential to reduce unfairness in today’s world—if people can agree on what “fairness” means.

“There’s a lot of pushback against AI because it can promote bias, but humans have been promoting biases for a really long time,” said psychologist Rhoda Au, PhD, a professor of anatomy and neurobiology at the Boston University Chobanian & Avedisian School of Medicine who also spoke at CES on harnessing AI ethically. “We can’t just be dismissive and say: ‘AI is good’ or ‘AI is bad.’ We need to embrace its complexity and understand that it’s going to be both.”

With that complexity in mind, world leaders are exploring how to maximize the benefits of AI and minimize its harms. In 2023, the Biden administration released an executive order on Safe, Secure, and Trustworthy AIopens in new window, and the European Union came close to passing its first comprehensive AI Actopens in new window. Psychologists, with their expertise on cognitive biases and cultural inclusion, as well as in measuring the reliability and representativeness of datasets—have a growing role in those discussions.

“The conversation about AI bias is broadening,” said psychologist Tara Behrend, PhD, a professor at Michigan State University’s School of Human Resources and Labor Relations who studies human-technology interaction and spoke at CES about AI and privacy. “Agencies and various academic stakeholders are really taking the role of psychology seriously.”

Government officials and researchers are not the only ones worried that AI could perpetuate or worsen inequality. Research by Mindy Shoss, PhD, a professor of psychology at the University of Central Florida, shows that people in unequal societies are more likely to say AI adoption carries the threat of job loss (Technology, Mind, and Behavior, Vol. 3, No. 2, 2022opens in new window).

Those worries about job loss appear to be connected to overall mental well-being. For example, about half of employees who said they were worried that AI might make some or all of their job duties obsolete also said their work negatively impacted their mental health. Among those who did not report such worries about AI, only 29% said their work worsened their mental health, according to APA’s 2023 Work in America survey.

“In places where there’s a lot of inequality, those systems essentially create winners and losers,” Shoss said, so there is additional concern about how AI tools could be used irresponsibly—even maliciously—to make things worse.

Those fears are not unfounded. Biased algorithmic decision-making has been reportedopens in new window in health care, hiring, and other settings. It can happen when the data used to train a system is inaccurate or does not represent the population it intends to serve. With generative AI systems, such as ChatGPT, biased decision-making can also happen unexpectedly due to the “black box” issueopens in new window, which refers to the fact that even an algorithm’s developers may not understand how it derives its answers.

“Even if we give a system the best available data, the AI may start doing things that are unpredictable,” Luxton said.

Examples include a recruiting tool at Amazon that preferred male candidates for technical jobsopens in new window, and Replika, an AI companion that harassed some of its usersopens in new window. Avoiding such issues requires careful auditing of AI tools—including testing them in extreme scenarios before they are released—but it also requires significantly more transparency about how a given algorithm learns from data, Luxton said.

On top of technical audits, Behrend and Richard Landers, PhD, a professor of industrial-organizational psychology at the University of Minnesota Twin Cities, have published guidelines for conducting a “psychological audit” of an AI model, or evaluating how it might impact humans (American Psychologist, Vol. 78, No. 1, 2023opens in new window). That includes direct effects, such as who is recommended by a hiring algorithm, as well as broader ripple effects on organizations and communities.

The audit employs basic principles of psychological research to evaluate fairness and bias in AI systems. For example: Where is the data used to train an AI model coming from, and does it generalize to the population the tool intends to serve? Were the data collected using sound research methods, or were limitations introduced? Are developers making appropriate inferences from that data?

Conversations about algorithmic bias often center around high-stakes decision-making, such as educational and hiring selection, but Behrend said other applications of this technology are just as important to audit. For example, an AI-driven career guidance system could unintentionally steer a woman away from jobs in STEM (science, technology, engineering, and math), influencing her entire life trajectory.

“That can be potentially hugely consequential for a person’s future decisions and pathways,” she said. “It’s equally important to think about whether those tools are designed well.”

Even if an algorithm is well designed, it can be applied in an unfair way, Shoss said. For example, a system that determines salaries and bonuses could be implemented without transparency or human input—or it could be used as one of a series of factors to guide human decision-making. In that sense, using AI ethically requires asking the same questions that evaluate any other organizational change: Is it done with trust, transparency, and accountability?

An algorithm itself may be biased, but humans can also introduce inaccuracies based on how they use AI tools.

“AI has many biases, but we’re often told not to worry, because there will always be a human in control,” said Helena Matute, PhD, a professor of experimental psychology at Universidad de Deusto in Bilbao, Spain. “But how do we know that AI is not influencing what a human believes and what a human can do?”

In a study she conducted with graduate student Lucía Vicente, participants classified images for a simulated medical diagnosis either with or without the help of AI. When the AI system made errors, humans inherited the same biased decision-making, even when they stopped using the AI (Scientific Reports, Vol. 13, 2023opens in new window).

“If you think of a doctor working with this type of assistance, will they be able to oppose the AI’s incorrect advice?” Matute said, adding that human users need the training to detect errors, the motivation to oppose them, and the job security to speak up about it.

Decades of psychological research clearly show that once humans inherit a bias or encounter misinformation, those beliefs are hard to revise. Celeste Kidd, PhD, an assistant professor of psychology at the University of California, Berkeley, argues that assumptions about AI’s capabilities, as well as the way many tools present information in a conversational, matter-of-fact way, make the risk of inheriting stubborn biases particularly high (Science, Vol. 380, 2023opens in new window).

“By the point [that] these systems have transmitted the information to the person…it may not be easy to correct,” Kidd said in a press release from the university (Berkeley News, June 22, 2023opens in new window).

Companies also can—and do—intentionally leverage AI to exploit human biases for gain, said Matute. In a study of simulated AI dating recommendations, she and graduate student Ujué Agudo found that participants were more likely to agree to date someone whose profile they viewed more than once, a choice she said is driven by the familiarity heuristic (PLOS ONE, Vol. 16, No. 4, 2021opens in new window). Guidelines for ethical AI should consider how it can be designed to intentionally play on cognitive biases and whether that constitutes safe use, she added.

“We all have cognitive biases, and AI can be used to exploit them in a very dangerous way,” Matute said.

While poorly designed algorithms can perpetuate real-world biases, AI may also hold the power to correct or reverse inequities among humans. For example, an algorithm could detect whether a company is less likely to hire or promote women, then nudge leaders to adjust job ads and decision-making criteria accordingly.

“There are risks here, too, and it’s equally important to have transparency about these types of systems—how they’re deriving answers and making decisions—so they don’t create distrust,” Luxton said.

Using AI to reverse bias also requires agreeing on what needs to change in society. The current approach to building AI tools involves collecting large quantities of data, looking for patterns, then applying them to the future. That strategy preserves the status quo, Behrend said—but it is not the only option.

“If you want to do something other than that, you have to know or agree what is best for people, which I don’t know that we do,” she said.

As a starting point, Behrend is working to help AI researchers, developers, and policymakers agree on how to conceptualize and discuss fairness. She and Landers distinguish between various uses of the term, including statistical bias versus equity-based differences in group outcomes, in their recent paper.

“These are noncomparable ways of using the word ‘fairness,’ and that was really shutting down a lot of conversations,” Behrend said.

Establishing a common language for discussing AI is an important step for regulating it effectively, which a growing contingent is seeking to do. In addition to Biden’s 2023 executive order, New York State passed a lawopens in new window requiring companies to tell employees if AI is used in hiring or promotion. At least 24 other states have either proposed or passed legislation aiming to curtail the use of AI, protect the privacy of users, or require various disclosures (U.S. State-by-State AI Legislation Snapshot, BCLP Law, 2023opens in new window).

“It’s pretty difficult to stay on top of what the best practice is at any given moment,” Behrend said. “That’s another reason why it’s important to emphasize the role of psychology, because basic psychological principles—reliability, validity, fairness—don’t change.”

Luxton argues that executive orders and piecemeal legislation can be politicized or inefficient, so policymakers should instead focus on establishing standard guidelines and best practices for AI. That includes requiring developers to show an audit trail, or a record of how an algorithm makes decisions. (Luxton is also writing a guidebook for behavioral health practitioners on integrating AI into practice.) When challenges arise, he suggests letting those play out through the judicial system.

“Government does need to play a role in AI regulation, but we also want to reduce the inefficiencies of government roadblocks in technological development,” Luxton said.

One thing is clear: AI is a moving target. Using it ethically will require continued dialogue as the technology grows ever more sophisticated.

“It’s not entirely clear what the shelf life of any of these conversations about bias will be,” said Shoss. “These discussions need to be ongoing, because the nature of generative AI is that it’s constantly changing.”