Kim Mills: Think about the sweetness of a strawberry or the satisfying crunch of a potato chip. What makes these things taste so good? You might think it’s just sugar, salt, and fat, but research suggests that it’s more than that. The color of the strawberry, the sound of the potato chip crunch, even the place where you’re eating can all contribute to how you experience each bite. Psychologists have long studied the senses individually, but in the real world, sight, sound, taste, smell, and touch are constantly interacting.
Today we’re going to talk to a psychologist who studies multisensory perception about how our senses work together to shape how we perceive and taste the world. So how does the brain combine information from different senses to create our global perceptions? What are some of the most surprising ways that our senses interact? And how can you use this knowledge to eat healthier, enjoy more food, and otherwise improve your life?
Welcome to Speaking of Psychology, the flagship podcast of the American Psychological Association that examines the links between psychological science and everyday life. I’m Kim Mills. My guest today is Dr. Charles Spence, a professor of experimental psychology at the University of Oxford, where he heads the Crossmodal Research Laboratory. He’s published hundreds of academic papers on multisensory perception. He also brings his knowledge out of the lab and into the world. He’s consulted for many multinational companies on product design, packaging, and branding. He’s worked with chefs to design foods that maximally stimulate the senses and he’s helped car companies design multisensory warning signals for drivers. Dr. Spence has won many awards for his work, including a 2008 IG-Nobel Prize, for research that makes you laugh and then makes you think. He’s also author of the books Gastrophysics: The New Science of Eating and Sensehacking: How to Use the Power of Your Senses for Healthier, Happier Living.
Dr. Spence, thank you for joining me today.
Charles Spence, PhD: Thank you.
Mills: So, I just mentioned that you direct the Crossmodal Research Laboratory at Oxford, so let’s start there. What does “crossmodal” mean in your research?
Spence: So, as a psychologist who studies the senses—sound, touch, taste, and smell—I spend most of my time thinking about how the senses relate one to another. And the two words that we use regularly are crossmodal on the one hand and multisensory on the other. Many people use those terms interchangeably, but to those who’ve spent their lifetime thinking about the ways that the senses can interact, multisensory is taken to mean when things come together into one unified gestalt or whole. So, maybe the flavor of food has taste and it has smell, texture. All of those things come together into one unified flavor, which is multisensory. I contrast that with sort of crossmodal cases where, for example, if I suddenly slapped my hands together, that sudden loud noise will make the light seem brighter wherever you are. You don’t integrate that clapping sound with the lights, but nevertheless, the sounds crossmodally affect visual perception. So, the things kind of stay separate and yet one influences the other.
And over the last 30 years, I’ve spent a lot of time thinking about these crossmodal and multisensory effects, starting in the lab and then going out into the real world to see where they apply and then coming back into the lab again to study them some more.
Mills: All right. Well, we’ll get around to talking about that, but I want to start by noting that most of your work has focused on food and taste. Why is eating such a powerful example of multisensory perception?
Spence: So, I guess eating and drinking is something that we all do. So, it’s a shared interest. It’s something we all do several times a day normally and it is probably one of the most multisensory things that we do. I mean, watching TV, that’s got sort of sound and it’s got sight, but smell-o-vision’s never gone very far and we can’t taste the TV yet. So, it’s multisensory but only with a few; whereas, food and drink kind of inherently stimulate all of the senses. They have a certain visual appearance that might look good on Instagram. They have the sound of the crunch or the crackle or the snap or the crack or the pop to stimulate our ears, the mouth feels of oral touch, the texture of food, the spiciness—another sense of pungency of pepper, the burn of chili—what have I missed there? The smell of it and the basic tastes of sweet and sour and salty and bitter. And all of those things, not every time, but very often all of those senses are kind of brought together to contribute to what we think of, what we describe as a taste of food, but really, we mean the flavor of food or the flavor. “I love the flavor of that” would be much better descriptive than I love the taste of it, because taste to the food scientists would narrowly mean just what you get on your tongue, which would be sort of sweet, sour, bitter, salty, umami, maybe a few other things.
Mills: Now, you found that sound can influence taste, something that you call sonic seasoning. How does that work? And maybe you could explain your sonic chip experiments as part of the answer.
Spence: Yes. Fortunately or unfortunately, my father’s no longer with us, but he would always, when he heard me talking about the sonic chip, he would go not the sonic chip again. He was tired of hearing about it, but maybe not everyone has. And this was work we did back in 2004, got the IG Nobel Prize for nutrition in 2008 for a study where we brought 20 hungry undergraduates into the lab in Oxford, the Crossmodal Research Lab, and gave each of them two tubes of Pringles to munch on and put them in a soundproof box with a microphone. And each time they bit into one of those Pringles, we’d take the sound, pick it up from the microphone, take it outside, transform it a bit, make it louder or quieter, boost just the high frequencies or suppress them in the crunching sound and then play them back over headphones. And we were able to show that we can enhance the crunchiness, the crispness, and the freshness—and hence, the liking of those potato chips—by about 15% simply by enhancing the sound of the crunch.
We never thought anything of it at the time, just published it in some obscure journal, but then it sort of got picked up by the IG Nobel committee as why on earth would anybody study that? And maybe I should get a second IG Nobel because I’m still studying it 25 years later, and it illustrates for us the example of sound as a forgotten flavor sense. The sound of the crunch. The sound of the food itself. But really, turns out there is sort of a mystery—as a title of on article in Science in 2000 from Weiss put it—why does nobody like a soggy chip, or potato chip or crisp as we call them? It’s got the fat, it’s got the flavorings, it’s got the salt. The only thing it doesn’t have is the crunch, but crunch noise in food has no nutritional value. So, why are so many of our snack foods crispy; crunchy; crackly; snap, crackle, and pop is sort of a mystery.
And from there, we’ve been looking both at the sounds of freshness and maybe in fresh produce, like apples and celery and carrots, allowed a snap or crunch when you bite in, it signals of fresher produce, which indicates greater preserved nutrient density from a fresher product. Maybe in the fried and baked category, the biscuits, the crisps, the breakfast cereals, maybe the sound of the crunch, maybe it indicates fat content, I wonder because maybe we don’t have fat receptors as such in our mouths, but fat’s something our brain loves for all the energy. So, maybe it’s a sound of crunch that we’ve learnt probably means it’s a fatty food, energy dense, and hence we like the sound.
Mills: It’s not just the food itself. You have found that even the plates and the cutlery we eat with can affect how we experience food. What are the factors that go into that?
Spence: Yeah. So, I guess my food research—I wasn’t initially interested in studying food because I was interested in the senses and I would be doing lots of warning signals work on cars and interface design, and food and flavor was just what food scientists did, not what psychologists did in my mind and many others. I got sort of dragged into food through Unilever, who were funding the lab at the time and that led into the sonic chip. And from there, we sort of started moving out, thinking about not just what sound does food make, but also what does it look like? Changing the color of fruit teas, say, in order to set people’s expectations for flavor. If you see a very bright colored tea, you expect it’s going to have an intense flavor. A lot of the products on the market 20 years ago didn’t, they looked great, they smelled great, but when you tasted them, they were kind of a bit thin because they didn’t have enough sugar in them as it turns out. So looking at the taste, the texture, the pungency, the sound of food, all stuff within the food itself and how the sensors interacted there.
And as the years have gone on, we’ve kind of moved out to think about the sound of the packaging, the color of the packaging, does that affect the taste? Yes, it does just as much as the, or maybe almost as much as the color of the food or drink itself. And then again, to the environments in which we eat and drink, thinking about the color, the lighting levels, the background music or soundscapes and they’re showing that all of these things, they’re not part of the flavor of food or taste, and yet nevertheless, they can systematically influence the taste. So, for example, we did an experiment with about 3,000 people, maybe one of the world’s biggest wine tasting experiments in London in 2014. And over four days, we invited these 3,000 people in as part of a food festival, gave them all a glass of red wine and a black tasting glass so they couldn’t see it, got them to rate the fruitiness, the freshness of the wine, how much they liked it or not under white lighting. Then we flipped the light switch, turned the room lights to red, asked them to rate the wine again, then flipped the room lights to green, rated the same wine again and then put the lights back to red, but with some sweet music—some sweet sonic seasoning in the background—and they rated the wine a fourth time. And amazingly, simply by changing the color of the walls, we were able to bring out the fruitiness of the wine by putting the red lights on by about 15%. Turn the lights to green, bring out the fresh acidic notes in the wine. And then, add some sweet or sour sonic seasoning, which sort of makes no sense at all. I mean, why would room color lighting, why would music sonically season, change the taste of the food? Because it’s not part of the food, and yet surprisingly, it does. And I’m still getting to the bottom of it.
I remember in 2008, we did the sonic chip and around that time, I got introduced to a chef called Heston Blumenthal. We put him in our booth, gave him some Pringles. He came out a changed man saying, “Wow, sound is a forgotten flavor sense. Sound’s an ingredient the chef can use in the kitchen.” And that led into a dish that was then served in the restaurant, in the world’s best restaurant as a signature dish, the sound of the seafood dish that comes to the table looking like the seashore with sand and sashimi foam seaweed laid out, but the waiter brings a conch shell out of which dribbles some earbuds and recommends that the diner put those earbuds in, where upon the diner hears the sounds of the sea, the seagulls, the waves gently crashing on the beach, but that was sort of built on research we did together with the chef to show that these seaside sounds do make seafood taste better.
And the chef has creatively taken that insight and then turned it into a dish but has since been sort of replicated around the world and hopefully gets diners coming out of the restaurant, if they’re lucky enough to go, thinking for the first time, “Yeah, sound is important and I never thought about that before,” and has this amazing effect that, for some number of journalists who write about going to the restaurant and even for people who are lucky enough to get their 50th birthday party for a once in a lifetime meal, they come away and they say, “I cried.”
Mills: You work with a chef who had this meal, I think he was in a restaurant in Milan—and we’re talking about making people cry. I mean, when you got to the—it was a multi-course meal—when you got to the sixth item, which was rice pudding. How can rice pudding make people cry? I mean, what was it about that particular dish, the way that it was all put together?
Spence: So, going back to this sound of the sea dish that came onto the menu in the world’s best restaurant in 2007, had no idea it would make anybody cry, but as the newspaper reports came in about this dish and we saw it was happening, I’ve been sort of collecting examples of these sort of extraordinary responses to food, dishes that make people cry or that are inspiring or magical or give you an ASMR tingle. And just by chance, I got introduced to Federico, the chef in Milan, his restaurant Sensorium. He’s a young chef. Every year, he comes out with a new menu, 10 courses thereabouts, and you all sit at a counter in front of the chef, and for this Ayahuasca-themed menu, all those sort of mystical drugs of the Amazon, without the drugs, was this concept.
And on course number six, as you say, when he started serving this Ayahuasca-themed menu, people started crying and it was not what he intended either. It just sort of happened and they never cried for any of his previous menus and don’t cry on the first course or any other courses on this menu. It was only course number six, the rice pudding dish. And when I got to hear about that, I was lucky to get to eat at his restaurant, experience a thing for myself. I didn’t cry, I have to say. And then, we got talking to the chef and have been able to do experiments in his restaurant with his diners on that dish to figure out what’s going on.
I think nobody cries in a restaurant when you first walk in the door. You need to relax and feel comfortable in the space. So it’s probably that this course six out of 10 is important. You’re relaxed by this point and the chef kind of creates an emotional journey because it’s like two and a half hours, three hour meal, so a bit like a movie, and you have all these emotional trajectories of storytelling and this menu is on the kind of Cinderella thing he describes. Things get good and then they drop down to really bad and chaotic. And it’s just after that really bad moment, that’s when this rice pudding occurs. So, on this emotional trajectory, once you’re relaxed, that’s when it happens and the dish comes to the table as rice pudding, the waiter, waitress will break a wafer over the top of the bowl for you. So, I wonder whether that’s kind of the Eucharist sort of religious rituals going on there maybe and as well there are sounds, there’s some music and there’s like a nostalgic sounds.
There’s an old man recounting his life from one loud speaker. There’s a woman on the other corner like a school mistress with little children running around in a school bell. And then at the same time, there are also some very low frequency vibrations like sub audible naught to 20 hertz sound waves, but you can’t hear them, but they make your body feel something like tingle almost. You’re aware of your body for more than normally the case. And so with the chef, we’ve been giving a hundred people the full experience asking them whether they cried or felt like crying. Did those who cried rate the food as tasting better? It’s kind of important, which they did. And then another hundred diners without those low frequency vibrations that make you aware of your body and then a hundred delivery drivers who walk up with this week’s tomatoes, and they say so would you try some rice pudding? Yes. Did you feel like crying? What are you talking about? I have no idea. Don’t be stupid. So, none of the delivery drivers cried. Suggesting it’s not just about the rice putting itself. It’s something about the whole kind of gestalt of the experience and that when we turn those bodily vibrations off, the low, ultra low frequency vibrations, then the rate of crying drops significantly, suggesting that’s kind of a part. So, we think it’s probably these vibrations make you aware of your body and when you’re aware of your body’s sensations and you’re in this sort of nostalgic emotional frame, then it all kind of comes together and just the tears start flowing for two-thirds of diners.
Mills: That’s amazing. But that’s the kind of thing where you can only do it in the real world, right? I mean, you can’t really recreate something like that in a laboratory.
Spence: Absolutely. And it’s kind of one of the funny things that these days as psychologists, we always have to fill out endless ethics forms before we undertake research. And if my ethics panel got to hear that my subjects cried, my participants, sorry, these days, cried, I’d been in big trouble. What on earth are you doing? How dare you do that? It’s not ethically justified. Whereas the creatives, I think I know artists, but also the chefs, mixologists, they’re both perhaps more capable of eliciting these extreme strong emotions and then dealing with the consequences with which they have the tissues on hand. And so yeah, this is the sort of thing that works much better there and you probably couldn’t recreate in the lab.
And I sort of remember when we were doing the sound of the sea experiments, the chef came up to Oxford, to the science museum here, and we had 150 people for a conference. They’d pay somebody that wanted some food. He brought up a van of oysters. We did the sound of the sea versus the sound of farm yard chickens; showed that the sea sounds made things better. And then, he also brought up some bacon and egg ice cream, which was a dish in the restaurant at the time, very strange bacon and egg and ice cream. It’s kind of all kinds of wrong, but sort of molecular and modernists gave people that to eat while they heard the sounds of sizzling bacon or the farm yard chickens, we could flip the flavor and then that’s sort of great. The dish works very well in the restaurant, but then we had some more batches of the bacon and eggs ice cream that I would give to participants in the lab for some follow up, carefully controlled studies and there people go, “Oh, it’s horrible in the wrong context.” It’s like, that’s just really nasty ice cream. If you know it’s like a designed experience from the creative chef or in a design shop, then you’re much happier to, you may even expect to be surprised or confronted by sensory incongruency. So, there it works, but only there can you study it and things like the surprise when you bite into something that looks like an apple that tastes like a pineapple or that you can only do once. And again, that’s very hard to capture in a lab situation when somebody comes in a white coat and gets you to fill in the consent form and is monitoring you and the whole atmosphere is not conducive to maybe that more, how should we put it, emotional or engaged response or something.
Mills: A few years back we, on Speaking of Psychology, did an episode on synesthesia in which the researcher, Dr. Julia Simner, talked about how people with synesthesia can taste certain words, which can even taste nasty. She said she had one subject who thought her name was nasty and so he wouldn’t even say it. So, how is what you study different from synesthesia?
Spence: So, there are all these things, like the crunch of the crisp, and it’s all kind of part of the thing itself. That’s sort of maybe multisensory. And then, there are these more surprising connections between the senses, such as when I ask you to think of what’s the sweetest color, most people will say pinkish red or something. If I ask you to say, is sweetness round or angular? People think it’s going to be sweet is round, bitterness is angular. And these are like these surprising connections between the senses and that’s a bit like synesthesia, but it’s, I think, they’re fundamentally different and Jules—who I was an undergraduate with here in Oxford back in the ‘80s—we have debates about this. I think synesthesia’s for a long time played a really important role in stimulating artists, creatives—Kandinsky, Scriabin, you name it.
But the synesthetic connections are kind of unusual. They’re idiosyncratic. For one synesthete the experience is blue for Monday. The next synesthete says, no, it’s orangey, gold. And so, the connections are different. Whereas, what I’m studying is this kind of universal synesthesia, which I’ve called crossmodal correspondences—these surprising connections between the senses that we share between tastes and colors and shapes and textures and sounds and timbres and tempos and rhythms and roughness—very surprising, but really interesting. And so, in our research, after we did the sound of the sea experiment that led into the dish, we’d never have done that study if there hadn’t been 150 people turning up the next day and they demanded some food, we didn’t think of what to do. It’d be too crazy to think that playing seaside sounds would change the taste of—after a few years of the dish being on the menu, people started saying, “Well, it’s obvious, isn’t it? Seaside sounds, fish and chips taste better by the seaside. We didn’t need your psychologist to tell us that.”
So, we had to try and think of the next new thing and that became this sort of sonic seasoning narrowly defined, which is when we use music to change taste of food because if we know that sweet tastes are more high pitched, more likely to be piano sounds, bitter tastes, that’s for the lower the pitch of the music or sound, the more bitter brassy sounds are bitter. And now, we have these whole musical menus for bitter, spicy, creamy, sweet, salty, umami of exactly what the music or soundscape should have in terms of tempo, rhythm, and so on. And then, we can either find music off the shelf that has those qualities or get a composer or sound designer to create compositions to match, accentuate the taste. And this is often described as synesthesia. Certainly back in the 1930s, the Italian futurists, people like F.T. Marinetti, he’s got a whole futurist cookbook where you find most of modernist cuisine in idea, if not an execution. And probably a horrible man, sort of misogynist and all sorts of bad things, but in the future cookbook, there are all sorts of great ideas and he talks about syntactilismo, sin auditivo—synesthetic sound, synesthetic touch. They thought it was synesthesia then.
I think they were wrong about that, but they were right that textures and tastes correspond. We can study them in the lab and then work with chocolatiers, wine brands, coffee, designers, platewear manufacturers such that we can hear by changing the texture on your coffee cup to be rougher or smoother. We can then change your experience of the body of the coffee, which could be a good thing or a bad thing, depending on the brew you’re drinking and what kind of taste world you live in.
Mills: So, do you do dinner parties where you curate the music to go along with what you’re serving?
Spence: Absolutely, of course. Certainly we do. I have to contend with Mrs. Spence, so a lot of my ideas about gastrophysics, I’m not allowed. All my black angular plates, I’ve got a very good scientific reason why they work for certain foods, but she doesn’t like them, so they’re out on the street. A few things stick though, but once a month she will let us have a multisensory lab dinner party and then we’ll get to have some fun. We’ve had some chefs in residence over the years and they would cook away. I remember one memorable occasion where we had Rupert Ponsonby who does beer marketing and he brought a whole range of beers and then we had my wife’s ironing table all the beers arranged on top and a tone generator and we sort of recreated Hansen’s 1969 pitch of harmony experiment when he brought people into Copenhagen University, gave them Carlsberg regular lager and Carlsberg elephant export lager that’s stronger and said, “Just twiddle with the tone on the tone generator totally find the frequency that matches the taste.”
People think it’s bizarre, but then with 10 people and sort of 10 bottles of beer, you find that people match the stronger, more alcoholic export lager to a higher pitched sound. And so, we recreated that. And again, he in 1960s and ‘70s was showing that when you played the pitch of harmony to people as they tasted the matching beer, then they had these sort of experiences where the hands were shaking, their heads were expanding—the kind of research that I guess made psychologists not ask their participants what they thought anymore. Just fill in the question, just give me a one to 10 answer. That’s all I want to hear from you. Whereas, in the old days, you’d ask people, how is it for you as a subject? What went on? And you got all these weird responses. And ever since I first came across that study in the library when I was an undergraduate, I wanted to reproduce it and we had it there in the lab dinner while the chef was cutting potatoes into 3D sculptures, just like Marinetti was doing in the 1930s.
And he’s turning my wife’s Christofle, very expensive silver cutlery, and he’d been to the market at the chef and he’d bought, I’m not sure, a bunch of hares with their pelts and they wrapped the skin and the hare, the pelt of the hares, around my wife’s cutlery and then we served a hare stew and anybody who’s there will still remember it. That you’re sort of feeling the animal, the warmth of the fur in your hand or the spoon. You could smell the animal and you’re eating it, and it makes you much more mindful about what you’re eating. And it is kind of a syntactile, as most kind of touch and taste coming together just like in those futurist dinner parties. And yeah, lots of fun was had by all and amazing how many, that’s just like a chance to play really and to see what works and some of the dishes that have been served in those multisensory lab dinners then end up being kind of formal research that we follow up in the lab with larger sample sizes who are blinded to the purpose of the research and so on and so forth.
Mills: I want to ask you—a lot of us, particularly in the States, maybe you do this in England as well, a lot of us eat in front of the TV. What is the impact of eating in front of the TV on how you experience your food?
Spence: Mostly bad, bad, bad from the Gastrophysics book. That was one of the strongest relations that came out was from research showing that we eat significantly more when distracted by technology. Quite how much more? It could be as much as 30% more when eating with the TV on than off. It all depends whether it’s a new show or a repeat and how entertaining the program is as to how much more, but yeah, significantly more with the TV on. And these days, I guess a lot of people will eat with their mobile device instead and there’s something like 10–15% increase in consumption. So, I think generally very bad, in part because those technologies just distract us from what we’re eating and I think our brain is going to use the sensations associated with consumption to figure out when we should stop eating. So, if all the sensations we get are from the TV of our mobile device, we’re not aware of the tastes and the textures. Then it takes more food before our brain says, “Okay, I’ve had enough food sensations. It’s time to stop now.” And that’s kind of why you end up eating more.
That’s definitely the bad side, but I do think there’s sort of potential here to turn it to a force for good sometimes. So, we did a project back in 2017 with one of the home food delivery companies here in the UK and I’m not sure whether it was Netflix or one of the other streaming services. We knew it was Game of Thrones. There was like a mafia TV series at the time and some other shows, and we knew what was going to be airing the next episode. So, worked with a chef to make the meals that match what the characters on the shows you might be watching on the streaming service would be eating. And there, I think, there could be a possible sort of resonance that you are almost, by eating the same thing as the characters on screen are eating, then it sort of connects you and maybe rather than distracting you, turns it into a more immersive engaging multisensory experience than otherwise would have been the case.
Mills: So, you were recently quoted in an article talking about Dubai chocolate, which is everywhere right now. What is it about Dubai chocolate that in a multisensory sense has made it so trendy?
Spence: I think it’s all about the eye appeal, the Instagram-ability, not really anything to do with the taste or the texture or the chemicals. You might like it or not. For me, it’s too sweet. It’s kind of this sort of chocolate on the outside, maybe wrapped as if it’s a gold bar, a bling appeal. And on the inside, I’ll have this very bright green kind of pistachio and other stuff, kind of crunchy interior. Didn’t exist 5 years ago and sort of exploded so that wherever you go now, like in a European capital, you’ll see that green and brown imagery for smoothies, for cakes, for hot drinks, for ice creams, for chocolate, for you name it. So, what’s going on? I think it’s really all about Instagram-ability in that if I discovered a new regular chocolate bar that was the best tasting chocolate I’d ever had, I couldn’t share that with you over Instagram or social media because the picture of it would just look brown like any other chocolate.
So, it’s impossible to kind of communicate visually, digitally deliciousness; whereas, the Dubai chocolate has this distinctively different eye appeal, the green and brown. That color combination doesn’t really exist in anything else. And hence, when I see it, I can uniquely index that food and hence it’s much more Instagrammable, and I think successful, paired with other things. Like Aperol spritz has had a moment in recent years and province or rose wine has had a resurgence of popularity in Europe. And again, for both of those drinks, the idea is they look good on Instagram or they look distinctively different. If you saw a clear drink, it could be gin and tonic, it could be Sprite, it could be soda water—you wouldn’t know what it was, but that kind of orangey of apple on a summer’s evening, somewhere in a piazza and a Mediterranean, then it can only be one thing. And it’s that distinctive difference that helps these products to really become viral and probably also explains then, if you have a product that doesn’t look distinctly different—the question of why things like extreme spicy challenges, extreme sour challenges are becoming a thing, and there I think it’s, why those two tastes? Well, because those are things that I can express through my facial pain. When eating a very hot chili food, you will see me sweating and a very sour food will make my face pucker up. And so, in a way, I can communicate those tastes through the extreme response in my face in a way that I couldn’t with sweetness. So, it’s going to look different or evoking you, the consumer, some extreme physical response that can then be conveyed through media and that these things, I think, are becoming more and more rapidly emerging as AI health starts to pick up much more rapidly on emerging food trends around the world—matcha tea or bubble tea or the Dubai chocolate—picking up on the trends rapidly and allowing food companies to deliver to market their new version of Dubai Chocolate X in time to meet the ascending wave of popularity.
Whereas, in the past, by the time the food company had heard about this new trend, it was kind of too late already to ride the wave.
Mills: Speaking of AI, do you have any thoughts on how and whether AI is going to be, say, designing new dishes or entire cuisines?
Spence: Yes and no. I think not in a way that so far people have thought about it, which is there’s been a lot of excitement about the idea of flavor pairing. The idea that you could combine ingredients effectively in a dish if you know what molecules and volatile compounds, chemicals they have in them and two foods that share similar chemicals will pair well together. So, you get things like caviar and white chocolate share a molecule and sort of jasmine and pork share a molecule. So, they go very well together. Cauliflower and dark chocolate or maybe it’s blue cheese, they go together. So, you get some very unusual combinations that work well and the idea being they share a molecule—and with AI and large language models, that seems to make the whole process of pairing flavors based on shared compounds much easier than looking up numbers in charts and things.
And yet, I think that’s not going to work in the end because I don’t think flavor pairing is generally a sound principle and certainly when people have studied it systematically foods are not likely to go well together just because they share a compound. But instead, I think the way that AI is already starting to change the food landscape is with, illustrated by a chef we work with in London called Joseph Youssef, and he very often gets invited to make new foods. For a virgin galactic over in Arizona when he had a party for the people who put a deposit down being on the first space tourist. Or when the Willie Wonka movie came to London for his launch with Timothy Chalamet, whatever a year ago, he was asked in both cases, “Can you make some canapes to match the event?” Very often when he makes those things, then the client says, “Okay, now that’s too adventurous. Just give me the cheese and pineapple sticks.” And he’s wasted all that effort.
So, what he started doing, which is really clever, I think, is asking AI to generate the visual concept of a dish. So, he types in for the, say for the Wonka movie, he says, “Give me a chocolate hairbrush.” And then, AI comes up with this amazing sort of chocolate eclair with hair. If the client likes the idea from just from the visual image, then he’s got sign off and then he goes and figures out how to make it and sources from around the world ingredients. So, to get the hair on the chocolate eclair hairbrush, that was going to Turkey for some Turkish pizzani or something. They have some sort of food stuff that matches and that’s been very successful for him as a kind of aid to culinary creativity, rather than a replacement of that, maybe it’s really good at generating ideas, but any of those ideas that are generated by AI, given that AI doesn’t taste nor smell, have to be then sense checked by the real creative of the kitchen in order to make sure that they work or otherwise.
When we did a study last 2024, we got lots of Instagram images of foods, raw foods, processed, and ultraprocessed, like carrots, carrot juice and carrot cake. And then we made a AI version of the same foods and asked whether people could tell the difference. And in 2024, with the software we used, people could say, “Yep, that’s AI generated food image." As such, we don’t like it as much as the real thing.” But roll the clocks forward now and with every evolution of the AI, it gets harder and harder to tell the difference and more and more tempting for the food companies in the marketing to present AI generated dishes.
And the real concern for me is that maybe they will be generated to be ultimately appealing to our brain because the AI will realize that if it’s the image of the food should be from a first person perspective and look large and have yolk porn with protein and emotion dribbling out and it should having lots of energy density because that’s what our brains love and it will create these hyper delectable food images that, even if we do not eat the images that we see, they may nevertheless set the standards for what we think is acceptable in a meal. And that’s going to go all the way through to one of my favorite examples from the last year or two was a restaurant in Austin, Texas—Ethos, I think it was called, the restaurant. And you go online, it’s got these amazing images of croissants in the shape of hippopotamuses that look huge and then sort of flower arrangements of sushi, amazing—and the whole thing’s AI generated. There is no restaurant, there is no restaurant manager, he’s a bot and all these food images are, but they’re sort of so appealing.
Mills: Everybody wants to eat there.
Spence: Yeah, everyone wants to eat there. I know we’ve all been tricked by one of these sort of digital hallucinations and yeah, it’s going to become harder and harder to watch out for them.
Mills: So, just to wrap up, what’s next for you? What are you studying now? What are your big questions ahead that you want to answer?
Spence: Ooh, all sorts of stuff. Maybe one of the big ones is when we’ve tried over the last decade or two to create these multisensory experiences by adding an extra sense of music or lighting to something or touch, but it’s often been just trying to add another sense to kind of match with what’s already there. So, we pick a sound that matches the taste, that’s congruent with it. But, I’ve recently been looking a lot at the literature on sort of media studies and film music design, an area I haven’t come across before, but it’s own separate world. And there, to see all the clever ways that sort of movie directors and those who create the compositions for movie scores will use sound sometimes in sort of a counterpoint. Sometimes, they’ll have really violent scenes on screen, Quentin Tarantino and Stanley Kubrick, sort of famous examples where when they have ultra violence on screen, they might have singing in the rain or something, happy music as sort of counterpoint. So, could you have a culinary counterpoint? I don’t think anyone has tried that yet, having music that deliberately goes against something in the dish and makes sense only as a kind of composite. And then, these other examples of all these big words I have to look up and I forget now, where the music starts and then something happens on screen and then the music keeps continuing. It’s like the music doesn’t care about the world, doesn’t care about what’s going on. It continues regardless of these terrible things. And again, it could use that sort of notion in a dining concept and what would it mean? And I think there are lots of these ideas from this kind of mood music in film to think, could we translate that into a multisensory dining experience to create new kinds of things where we don’t just match what’s already there, that do something different, something that augments it, and that maybe, ultimately, we can sort of deliver this kind of idea of a multisensory gestalt or the total work of art being very ambitious that does sort of engage all of the senses and only make sense when all the senses come together. We’ll see if we ever get there, but yeah, a worthy challenge that’ll keep me going for a good few years.
Mills: Well, I want to thank you, Dr. Spence, for joining me today. This has been just amazing and now I’m hungry and I have to go find something to eat.
Spence: Thank you.
Mills: You can find previous episodes of Speaking of Psychology on our website at speakingofpsychology.org or on Apple, Spotify, YouTube, or wherever you get your podcasts. And if you like what you've heard, please subscribe and leave a review. If you have comments or ideas for future episodes, you can email us at speakingofpsychology@apa.org. Speaking of Psychology is produced by Lea Winerman.
Thank you for listening. For the American Psychological Association, I’m Kim Mills.
Professor Charles Spence is the head of the Crossmodal Research Laboratory at the University of Oxford. He is interested in how people perceive the world around them. In particular, how our brains manage to process the information from each of our different senses (such as smell, taste, sight, hearing, and touch) to form the rich multisensory experiences that fill our daily lives. His research focuses on how a better understanding of the human mind will lead to the better design of multisensory foods, products, interfaces, and environments in the future.
