Science Brief

Blindness: remapping the brain and the restoration of vision

Sensory substitution technology enables sight without visual input.

By Michael Proulx

Michael J. Proulx Michael J. Proulx is senior lecturer in psychology at the University of Bath and visiting senior lecturer in multimedia at Queen Mary University of London. His research focuses on several aspects of crossmodal cognition and multisensory processes with a particular interest in the impact of blindness on cognition. He received his BS in Psychology from Arizona State University and his PhD in Psychological and Brain Sciences from Johns Hopkins University. He is the recipient of a New Investigator Award in Experimental Psychology: Human Perception and Performance and a Science Showcase Award from the American Psychological Association, and was honored as a torchbearer for the London 2012 Paralympic Games.

Author website

We live in a visually abundant world, both natural and created by us: sunsets in the desert, the twinkling night sky, city skylines reaching to the clouds, the beautiful face of a loved one. In fact, the world is so full of competing information for such a visually-inclined species as Homo sapiens that we (and many other animals) must be perceptually selective. Even though the primate brain devotes over 50 percent of the neocortex to visual areas (Van Essen, Anderson, & Felleman, 1992), the hierarchical organization of the brain still requires selection of a subset of information for full processing and any eventual motor response. Attention is the core ability to select some aspects of the world presented to our senses for further processing, and thus to ignore other aspects (Egeth & Yantis, 1997; Proulx, 2007). Thus, we can characterize one end of the visual information processing spectrum as an embarrassment of riches (cf. Wolfe & Horowitz, 2004). What would the other end of the visual information processing spectrum be like?

Imagine being struck blind. What would you miss? I think of those sunsets, and the face of a loved one. There are practical things too, such as the ability to navigate a world built by humans for the sighted. Consider visiting a new city, and navigating its public transportation system. Even if the language is foreign, certain universal symbols, such as arrows or a color-coded map, might allow for efficient travel from one point to the next. According to the World Health Organization, over 39 million people in the world are blind; over 246 million have some form of impaired, or low, vision ("Visual impairment and blindness," 2012). The number of people visually impaired from age-related disorders, such as glaucoma and macular degeneration, is on the rise, with additional problems brought about by the onset of diabetes, which is also on the increase ("Diabetes, Type 1," 2012).

How might psychologists help blind persons navigate the visual world or, even more ambitiously, perhaps see again? Miracles and invasive implants aside, basic research in experimental psychology is serving as a hub for interdisciplinary research to find a way to restore vision. As a starting point for developing assistive technology, this requires an understanding of many aspects of psychology: the role of visual experience for cognitive development, how one sense corresponds to another, perceptual learning, spatial cognition, object recognition, neural plasticity and functional neuroanatomy. After the development of such technology, psychology is also important for assessing the usability of such technology, as well as its impact: does it improve mobility, well-being, safety and happiness?

The study of blindness provides the opportunity to discover the contribution of visual experience to cognitive and perceptual processes and the functional neural architecture for cognition. This is made most clear by differentiating between congenital and late blindness; in this way it is possible to assess the role of vision during the development of cognition and perception. It has long been observed that the absence of vision appears to enhance processing of the remaining senses (James, 1950). Many empirical studies have reported results in support of this view, showing that blind persons can have superior perceptual discrimination and localization, verbal processing and memory capacity, as noted in a recent review (Pasqualotto & Proulx, 2012).

Remapping the brain

How does the blind brain support such superior processing? There is substantial evidence for the recruitment of the so-called “visual” cortex to support these behavioral enhancements (Pasqualotto & Proulx, 2012). A number of neuroimaging studies discovered that the occipital lobe of blind individuals is active during the processing of auditory, haptic and olfactory stimuli (Amedi, Raz, Pianka, Malach, & Zohary, 2003; Kupers & Ptito, 2011; Rombaux et al., 2010). These correlations imply that the enhanced abilities of blind individuals arise either by the recruitment of the otherwise dormant visual cortex by the remaining modalities or perhaps by the unmasking of visual cortical activity that normally supports such functions. The correlated brain activity in the neuroimaging studies has been buttressed by findings from studies of short-term lesions (induced by transcranial magnetic stimulation) and real lesions (stroke damage) of the occipital lobe, which show that the activity in the visual cortex is causally relevant (Amedi, Floel, Knecht, Zohary, & Cohen, 2004; Merabet et al., 2009).

The recruitment of visual areas has been found to be more common for congenitally blind individuals. This suggests that the early onset of blindness produces either extensive brain reorganization or more effectively unmasks the non-visual processing that takes place in putatively visual areas. The more likely hypothesis is that of unmasking preexisting activity. This view is corroborated by the effects produced by long-term blindfolding of sighted participants (Proulx, Stoerig, Ludowig, & Knoll, 2008) who start to exhibit brain activations and behavioral repertoires equivalent to blind people (Merabet et al., 2008). As a matter of fact, such rapid change in adult participants is not compatible with the establishment of novel neural connections, which therefore must already be in place.

This collection of results supports the idea of a metamodal (Pascual-Leone & Hamilton, 2001) or supramodal (Kupers, Pietrini, Ricciardi, & Ptito, 2011; Kupers & Ptito, 2011; Ricciardi & Pietrini, 2011) organization of the brain (Proulx, Brown, Pasqualotto, & Meijer, 2012). Although the brain has traditionally been subdivided into sensory-dominant regions, an emerging view is that brain areas would be better classified by the computations and tasks that are carried out. These results have also been supported by converging evidence by using sensory substitution. Interestingly, the sensory-neutral representation of shape has been validated with sensory substation devices that specifically use stimulation to sensory organs that are not normally used for the purpose of shape recognition or spatial navigation, such as hearing with the ears (Proulx et al., 2008) and feeling with the tongue (Proulx & Stoerig, 2006).

The impact of blindness on cognition

Although some aspects of non-visual perceptual processing are enhanced in blind individuals, divergent results have been reported in studies investigating spatial cognition. Spatial cognition has been tested in blind individuals for a suite of behaviors, including memory for arrays of objects lying within the manipulatory space (arm’s length), environmental knowledge and navigation. On the one hand, some researchers have reported results suggesting that congenitally blind people do not fully develop spatial cognition, and thus perform poorer than sighted and late blind participants (e.g., Pasqualotto & Newell, 2007). On the other hand, other researchers have reported that visual experience is not necessary for the development of spatial cognition and that blind individuals can perform spatial tasks at the same level as sighted (Landau, Gleitman, & Spelke, 1981).

We hypothesized that these conflicting findings could be resolved if another factor were considered that would classify these studies by the frame of reference required for the task (Pasqualotto & Proulx, 2012). Shelton and McNamara (2001) noted that objects can be mentally represented with respect to the position of the observer (an egocentric frame of reference) or with respect to the position of the object in the environment (an allocentric frame of reference).

Earlier research with sighted participants found that people have a preference for the allocentric representation (Mou & McNamara, 2002). If the objects are lined up in rows and columns, then we remember that format rather than how they looked from the view from an angle, for example. This is also true if you blindfold the participants and walk them from one 'home' location to each object in turn. Sighted participants will prefer the allocentric reference frame over the egocentric one. In our study we were curious whether having visual experience during child development is key to create the structures in the brain to support such an other-centred reference frame. So we tested people who were congenitally blind, participants who were sighted for some period of time and then became blind later in life and sighted participants. We blindfolded everyone and walked them to the locations of objects in a large room. Later we tested them on a computer with a virtual pointing task: "Imagine you are at the cup facing the book, point to the pan."

First we found an interesting difference between the congenitally blind and sighted people: Although the sighted people preferred the other-centred, or allocentric, reference frame, the congenitally blind participants preferred the self-centred or egocentric reference frame. The important piece of the puzzle, however, was whether the late blind people would perform like the congenitally blind, showing that current visual experience matters, or like the sighted, showing the role of early visual experience. The results were clear: The late blind performed the same as the sighted participants. Therefore having the experience of vision early in life lays the groundwork in the brain for the representation of locations in a different reference frame than that found in people who never had visual experience. We are now exploring the ways that other aspects of cognition that rely on the underlying function and structure of spatial processing, such as magnitude estimation, are also impacted by visual experience or the lack thererof.

Restoration of vision through sensory substitution

Recent research has demonstrated the power of neural plasticity in adults to allow one deprived of one sensory modality to receive that missing input through another, intact sensory modality. This is possible through a process known as 'sensory substitution' pioneered by Paul Bach-y-Rita starting in the 1960s (Bach-y-Rita, Collins, Saunders, White, & Scadden, 1969). Although the eyes seem to be a prerequisite for vision, 'sight' truly takes place in the brain. To allow sight to occur in the absence of visual input through the eyes, visual information can be transformed into information that can be processed as sound or touch, and thus give one the potential to see through the ears or tongue (Bach-y-Rita & Kercel, 2003; Proulx & Stoerig, 2006).

My group investigates sensory substitution with “The vOICe,” a visual-to-auditory sensory substitution device invented by Dr. Peter Meijer of the Netherlands (Meijer, 1992). The software converts images by taking a visual snapshot every second and scanning the columns of pixels from left to right. The vOICe maps visual images to sound via three primary dimensions (see Figure 1., adapted from Proulx et al., 2008): horizontal pixel location is coded by the time provided by the left-to-right scan of each image and by stereo panning; vertical location is coded by frequency, so that ‘up’ in the image is represented by high frequencies and ‘down’ by low frequencies; and pixel brightness is coded by loudness, such that a bright white pixel is heard at maximal volume, and a dark pixel is silent.

Figure 1. The hardware requirements for visual-to-auditory sensory substitution, and the conversion principles employed by The vOICe. Figure adapted from Proulx et al. (Proulx et al., 20082008).

Figure 1. The hardware requirements for visual-to-auditory sensory substitution, and the conversion principles employed by The vOICe. Figure adapted from Proulx et al., 2008.


A video demonstrating successful localization using The vOICe can be seen for a blindfolded participant in a recent study we conducted (Proulx et al., 2008) on using sensory substitution to find object locations. Note that this participant is using ‘spy’ sunglasses as a video input to the device and is completely blindfolded. Besides being able to carry out basic visual behaviors, such as determining what is where (Brown, Macpherson, & Ward, 2011), sensory substitution can also give rise to the phenomenal experience of vision (Proulx, 2010). Two long-term users of The vOICe who both became blind after having visual experience early in life report visual imagery that is automatically evoked by the device (Ward & Meijer, 2010), and thus experience a form of synaesthesia as sight restoration (Proulx, 2010): The auditory stimulation of the device evokes visual cortical activity (Amedi et al., 2007) that is associated with visual phenomenological experience.

Psychology as a core “hub” science between engineering and neuroscience

Most importantly, like all good science, there is a lot more to discover. How can we improve sensory substitution training to make it easier to learn to use The vOICe or other devices? Or perhaps, how can our knowledge of crossmodal correspondences (Proulx et al., 2012; Spence & Deroy, 2012) improve the device to better encode visual information in a way that is more naturally perceived in the other senses for those with sensory deprivation? The problems are not solely cognitive in nature; other areas of psychological science must contribute as well to assess whether such devices improve the quality of life of blind users, and perhaps assess if they make recipients happier or more productive. The implications go beyond psychology to address some of the greatest questions in science today, such as the nature of consciousness and sensory awareness (Proulx, 2011). These are all important questions for health and society, and psychologists are indispensable for answering them. A unified effort to understand the impact of blindness on cognition and brain function will ensure that the future is looking up for the development of innovative solutions for vision restoration.

Acknowledgements

This work was supported in part by grants from the Engineering and Physical Sciences Research Council, U.K. (EP/J017205/1), the Great Britain Sasakawa Foundation and a Marie Curie Intra-European Fellowship (PIEF-GA-2010-274163; to host Dr. Achille Pasqualotto). The author thanks his co-authors and participants who assisted with the work cited here and the inventor of The vOICe, Dr. Peter Meijer.

References

Amedi, A., Floel, A., Knecht, S., Zohary, E., & Cohen, L. G. (2004). Transcranial magnetic stimulation of the occipital pole interferes with verbal processing in blind subjects. Nat Neurosci, 7(11), 1266-1270. doi: nn1328 [pii] 10.1038/nn1328

Amedi, A., Raz, N., Pianka, P., Malach, R., & Zohary, E. (2003). Early 'visual' cortex activation correlates with superior verbal memory performance in the blind. Nat Neurosci, 6(7), 758-766. doi: 10.1038/nn1072 ]

Amedi, A., Stern, W. M., Camprodon, J. A., Bermpohl, F., Merabet, L., Rotman, S., Hemond, C., Meijer, P., & Pascual-Leone, A. (2007). Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nature Neuroscience, 10(6), 687-689. doi: 10.1038/nn1912

Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision substitution by tactile image projection. Nature, 221(5184), 963-964.

Bach-y-Rita, P., & Kercel, S. W. (2003). Sensory substitution and the human-machine interface. Trends Cogn Sci, 7(12), 541-546. doi: S1364661303002900 [pii]

Brown, D., Macpherson, T., & Ward, J. (2011). Seeing with sound? exploring different characteristics of a visual-to-auditory sensory substitution device. Perception, 40(9), 1120-1135.

Diabetes, Type 1. (2012): National Institutes of Health.

Egeth, H. E., & Yantis, S. (1997). Visual attention: control, representation, and time course. Annu Rev Psychol, 48, 269-297. doi: 10.1146/annurev.psych.48.1.269

James, W. (1950). The Principles of Psychology. Dover Publications.

Kupers, R., Pietrini, P., Ricciardi, E., & Ptito, M. (2011). The nature of consciousness in the visually deprived brain. Front Psychol, 2, 19. doi: 10.3389/fpsyg.2011.00019

Kupers, R., & Ptito, M. (2011). Insights from darkness: what the study of blindness has taught us about brain structure and function. Prog Brain Res, 192, 17-31. doi: 10.1016/B978-0-444-53355-5.00002-6

Landau, B., Gleitman, H., & Spelke, E. (1981). Spatial knowledge and geometric representation in a child blind from birth. Science, 213(4513), 1275-1278.

Meijer, P. B. (1992). An experimental system for auditory image representations. IEEE Trans Biomed Eng, 39(2), 112-121. doi: 10.1109/10.121642

Merabet, L. B., Battelli, L., Obretenova, S., Maguire, S., Meijer, P., & Pascual-Leone, A. (2009). Functional recruitment of visual cortex for sound encoded object identification in the blind. Neuroreport, 20(2), 132-138 110.1097/WNR.1090b1013e32832104dc.

Merabet, L. B., Hamilton, R., Schlaug, G., Swisher, J. D., Kiriakopoulos, E. T., Pitskel, N. B., Kauffman, T., & Pascual-Leone, A. (2008). Rapid and reversible recruitment of early visual cortex for touch. PLoS One, 3(8), e3046. doi: 10.1371/journal.pone.0003046

Mou, W., & McNamara, T. P. (2002). Intrinsic frames of reference in spatial memory. J Exp Psychol Learn Mem Cogn, 28(1), 162-170.

Pascual-Leone, A., & Hamilton, R. (2001). The metamodal organization of the brain. Prog Brain Res, 134, 427-445.

Pasqualotto, A., & Newell, F. N. (2007). The role of visual experience on the representation and updating of novel haptic scenes. Brain Cogn, 65(2), 184-194. doi: 10.1016/j.bandc.2007.07.009

Pasqualotto, A., & Proulx, M. J. (2012). The role of visual experience for the neural basis of spatial cognition. Neurosci Biobehav Rev, 36(4), 1179-1187. doi: 10.1016/j.neubiorev.2012.01.008

Proulx, M. J. (2007). Bottom-up guidance in visual search for conjunctions. Journal of Experimental Psychology: Human Perception and Performance, 33(1), 48-56. doi: 10.1037/0096-1523.33.1.48

Proulx, M. J. (2010). Synthetic synaesthesia and sensory substitution. Conscious Cogn, 19(1), 501-503. doi: 10.1016/j.concog.2009.12.005

Proulx, M. J. (2011). Consciousness: What, how, and why. Science, 332(6033), 1034-1035. doi: DOI 10.1126/science.1206704

Proulx, M. J., Brown, D. J., Pasqualotto, A., & Meijer, P. (2012). Multisensory perceptual learning and sensory substitution. Neurosci Biobehav Rev. doi: 10.1016/j.neubiorev.2012.11.017

Proulx, M. J., & Stoerig, P. (2006). Seeing sounds and tingling tongues: Qualia in synaesthesia and sensory substitution. Anthropology & Philosophy, 7, 135-151.

Proulx, M. J., Stoerig, P., Ludowig, E., & Knoll, I. (2008). Seeing 'where' through the ears: effects of learning-by-doing and long-term sensory deprivation on localization based on image-to-sound substitution. PLoS One, 3(3), e1840. doi: 10.1371/journal.pone.0001840

Ricciardi, E., & Pietrini, P. (2011). New light from the dark: what blindness can teach us about brain function. Curr Opin Neurol, 24(4), 357-363. doi: 10.1097/WCO.0b013e328348bdbf

Rombaux, P., Huart, C., De Volder, A. G., Cuevas, I., Renier, L., Duprez, T., & Grandin, C. (2010). Increased olfactory bulb volume and olfactory function in early blind subjects. Neuroreport, 21(17), 1069-1073. doi: 10.1097/WNR.0b013e32833fcb8a

Shelton, A. L., & McNamara, T. P. (2001). Systems of spatial reference in human memory. Cognitive Psychology, 43(4), 274-310.

Spence, C., & Deroy, O. (2012). Crossmodal correspondences: Innate or learned?, 3(5), 316-318.

Van Essen, D. C., Anderson, C. H., & Felleman, D. J. (1992). Information processing in the primate visual system: an integrated systems perspective. Science, 255(5043), 419-423.

Visual impairment and blindness. (2012): World Health Organization.

Ward, J., & Meijer, P. (2010). Visual experiences in the blind induced by an auditory sensory substitution device. Conscious Cogn, 19(1), 492-500. doi: 10.1016/j.concog.2009.10.006

Wolfe, J. M., & Horowitz, T. S. (2004). What attributes guide the deployment of visual attention and how do they do it? Nature Reviews Neuroscience, 5(6), 495-501. doi: 10.1038/nrn1411

The views expressed in Science Briefs are those of the authors and do not reflect the opinions or policies of the American Psychological Association.