Visual art is about seeing, and not just about what is seen. As Rudolf Arnheim (1969, p. 137) notes, “A picture is a statement about visual qualities,” which—we can note—occur because of the workings of the brain interacting with the world. Indeed, “The artist, after all, can only deal with those attributes of nature which his brain is equipped to see.” (Zeki, 1999, p.3) Thus, we can examine visual art in order to gain some understanding of the basis of human perception of the world. Vision science examines the functioning of the brain to gain an understanding of human perception of the world. Although the endeavors of the artist and scientist may seem to be of a different kind, attending to the work of both is essential when we try to understand how it is that humans respond to and construct their visual world. The use of the word ‘construct’ in the previous sentence stresses the fact that we do not ‘see what is out there’.
Rather, we build a momentary understanding in our head about the meaning of the light that our eyes receive, an understanding that results in an internal representation of the world based on previous experience and momentary needs. As Semir Zeki says in his wonderful book, “Inner Vision”, it is a “totally erroneous view than an image of the visual world is ‘impressed’ upon the retina and then transferred to be ‘received’ by the ‘seeing’ cortex, there to be decoded and analyzed.” (Zeki, 1999, p. 13) Such a view is the “myth of the seeing eye”.

Fig. 1
Further when we consider all of our knowledge of the processing that goes on in the brain we immediately recognize that “there are many differences between the retinal image and the perception it ultimately gives rise to, including the transmutation of a two-dimensional retinal projection into a three-dimensional perception of visual space, the addition of color (which doesn’t exist as such in either physical qualities of objects or the light that reaches our eye), [and] the perception of the world as stationary despite frequent movements of the image on the retina as a result of changes in the position of the eyes and head.” (Purves and Lotto, 2003, p. 1) That is, seeing is an act of imagination that tries to extract (and abstract) important qualities of the visual world so that we can use that information to better survive. We see some of what we need to see, some of what we expect to see, even some of what we want to see, but not in any sense do we see a veridical, three-dimensional image of what is ‘really there’. The human artifacts that we call visual art are empirical evidence that must be considered in understanding visual perception, perhaps even more so than the results of recordings made of the responses of neurons in the brain. Because seeing is an abstraction performed by the brain, it is informative to examine the abstractions rendered by artists. Art is the product of an individual who is doing more than just everyday looking. Rather, an artist is actively working to see. Thus, essences of what seeing involves are inevitably embedded in the art that is made. Therefore, analysis of art furthers our knowledge of the function and functioning of the visual system. (Artistic aesthetics is also an important topic, but beyond the scope of this essay.)
It is rightly noted that perception is a subjective experience and therefore I can never know what you see—for example, that you experience red or round the same as I do—however, I can examine the art you produce to get some clues to what is occurring in both of our heads. Recent books such Zeki’s “Inner Vision” and “Why We See What We Do” by Purves and Lotto (2003), informed by philosophical insights about ‘embodiment’ such as those of Lakoff and Johnson (1999) in “Philosophy in the Flesh” are pointing a way to an understanding of visual perception that in some way departs from previous approaches. This essay attempts to impart the flavor of this emerging integration.

Why do we see? Zeki puts it simply, “we see in order to be able to acquire knowledge about the world.” (Zeki, 1999, p. 4) We do so to guide our interactions with the world. How much of what is ‘out there’ do we need to see to accomplish this moment-to-moment need for survival? Often, not very much. Consider this rough drawing of the wall of a room and the door that leads out of it. The simple information about the discontinuities that are edges, which is all we see here, is enough to safely exit the room. We don’t need to know about color, texture or exact shape—just about the essence that is the bounded surface we identify as a door.

Figure 1

The cubists knew about such essences. Compare this section of a painting of Ambrose Vollard by Picasso and a photograph of the head of his subject.

Figure 2

Cubists, perhaps intuitively, realized that to represent an object’s essence accurately, they could abandon uniform perspective and lighting and move around the object, painting it from many different angles and lighting conditions, creating planes. This is not unlike a process that our brain uses when building images, but slowed down by the artist, almost to a momentary, iconic image. Cubists were hardly the first to do this. A portion of a painting (left) by the Japanese artist Hiroshige, made in 1834 is similar to the work of the cubists. I have used an abstracting filter in the software package Photoshop to give it even more cubist flavor (right), although the result suggests that such a transformation only modestly changes the original.

Figure 3

Cubism is hardly the only example of artistic seeing of essentials. Types of art such as pointillism, impressionism and expressionism, among others, also deal with and often isolate visual essences such as color, form, texture and light. However, we need not only consider the more abstract styles, because even representational art is doing just what the name implies, namely ‘representing’ the results of seeing. Realism is about more than just showing the real; while it represents, it does more than simply re-present (NOTE TO TYPESETTER: THE HYPHEN IN THE PREVIOUS WORD IS NECESSARY AND SHOULD NOT COME AT THE END OF A LINE AS IF IT IS SIMPLY HYPHENATING THE WORD AT THE END OF A SENTENCE).
It is a construction about light, color and all of the other basic attributes of the visual system. For example, this portion of the painting “Bucks County Barn” (1932 version) by Charles Sheeler seemingly makes no attempt to portray a farm structure in a surrealistic or abstract manner. Yet, the way light and color are used makes it similar to the planes constructed by Picasso.

Figure 4

Albert Bierstadt is credited with showing the then-isolated American West to people in the populous Eastern cities during the late 19th century. The majesty of his realistic landscape paintings inspired awe and a spirit of exploration. He did more, however, than paint what was there. His use of light is at the heart of the essence and excitement of many of his works, such as in this portion of his 1868 painting “Among the Sierra Nevada Mountains, California.”
Figure 5

The brain’s inner construction of visual ‘reality’ is temporal as well as spatial. This was brought out forcefully by the experiments of A.L. Yarbus (1959) who examined how eye movements direct the center of gaze around an object being viewed. He observed that from eye-movement to eye-movement, the viewer’s gaze centered on the eyes, nose and mouth, and much more infrequently on aspects of the facial periphery of the subject being viewed. If we add to this the knowledge that the area of fine discrimination of the retina, the fovea, ‘looks at’ about one degree of visual space (about the size of an adult thumbnail at arm’s length), an almost amazing result simply falls out. For two people in conversation separated by a typical distance, the fovea is ‘looking’ at an area on the face of the other that is about size of the orbit of one eye. This is the only part of the face at that instant this is seen clearly and precisely. The rest is indistinctly viewed with the low resolution of non-foveal retina. Further, the eye moves in abrupt jumps (saccades) only about 4 to 5 times per second, so in that time very little of the face is actually seen with good resolution. These five images (from left) show what is actually ‘seen’ clearly in about one second. The clear circular patch in each image is the region viewed by the fovea, superimposed on the blurry image meant to represent what is seen by the non-foveal retina.

Figure 6

Despite the incomplete, mostly blurred, disjointed images above, what we are certain we see is the picture at the right end of the sequence. This filling-in, internal sharpening of the blurred periphery, and construction of a whole, relatively stationary image is part of what has been called the ‘grand illusion’ of consciousness. In short, what you think you see is what you expect to see, based in large measure on your past experiential explorations of the world. You can find more demonstrations related to art, abstraction, construction and vision at my site “The Art of Vision”. How can we make sense of the way visual perception seems to be working? From about 1950, a major approach consisted of analysis of the results of visual psychophysical experiments involving human subjects, applying insights that accrued from recording responses of individual neurons in the visual system of non-humans such as the horseshoe crab Limulus, frogs, cats and monkeys. One example of this is the use of the phenomenon of lateral inhibition that is observed in neuronal response profiles as an explanation of the well-known edge-enhancing phenomenon called Mach Bands. (A demonstration of the Mach Bands illusion can be seen at many online Web sites where the lateral inhibition model is also discussed.) However, recent studies have started to cast doubt on the simplicity of such explanations, even for Mach Bands. (see Purves and Lotto, 2003, p. 72)
Figure 7

The study of Mach Bands is just one example of how careful analysis of visual illusions has informed our understanding of the neuronal processing that the visual system carries out. And, just as the original lateral-inhibition, neuron-based explanation of Mach Bands has been called into question, a collection of elegant experiments by Prof. Dale Purves and members of his group is causing a reinterpretation of many other illusions. Their work is beautifully illustrated at the and is forcefully presented in the new book “Why We See What We Do.” (Purves and Lotto, 2003). The main point made by Purves and coworkers is that “visual stimuli are profoundly ambiguous” and thus, “The primary problem with . . . any rule-based (i.e. simple, neuronal response-pattern based] scheme of vision is the sources of any retinal stimulus (and thus its significance for subsequent action) are unknowable directly. Any element of a visual stimulus could have arisen from many - indeed, infinitely many - different objects and conditions.” Thus, visual perception does not yield a faithful reproduction of what is ‘seen’ but instead is an internal construction. If so, what governs such construction? Purves and colleagues argue that it is based on a probabilistic matching of the overall visual signal with the sum of stored information of what has been previously seen and experienced. As the subtitle of their book says, this is “An Empirical Theory of Vision” that uses “the phenomenology of what we see as a means to explore the link between the sources of visual stimuli in the physical world and perception, (and, by the same token, the links between physics, neurobiology and psychology).”
And, I might add, art. Their book presents the results of nume rous, carefully designed, quantitative experiments they have conducted on human visual perception that support this approach. What does it mean to build up an internally stored collection of experience in seeing? Do we extract primary qualities and aspects, much as the planes seen by the cubists? Do we form gestalts of the kind and in the way put forward by the Gestalt psychologists such as Westheimer? Clearly, we cannot arbitrarily make the world up as we go along - that is an obvious path to failure (and to being prey instead of predator). It is useful to explore to what extent the way that we make sense of the visual world relates to our overall needs to sensibly approach the physical world around us. Insights offered by Lakoff and Johnson (1999) in “Philosophy in the Flesh” offer clues to a way of approaching this question of how internal construction comes to represent the world external to us. Lakoff and Johnson are well-known for their explanation of how we know and understand abstract concepts through the use of metaphor (see their book “Metaphors We Live By”, 1980).
Their approach is easily explained by example. I might ask you, the reader, at this point whether you see what I mean in the main arguments I have presented? I know that this article can’t be too long because we are all busy and time flies. I also hope that I am being clear enough so that you will grasp my meanings. Lakoff and Johnson (1999, pp. 50-54) note of the italicized metaphors in the previous sentences that they express the ideas that “knowing is seeing,” “time is motion” and “understanding is grasping”. By itself, this insight into metaphor is informative about how we construct and understand our relationship to the world around us. Their 1999 book goes further, however, and suggests how our primary metaphors are actually fundamental mental embodiments of our sensorimotor, bodily interactions with the world. So, they argue that “knowing is seeing” because of our primary experience from birth of “getting information through vision; that “time is motion” because of our primary “experiencing of the passage of time as one moves or observes motion”; and, that “understanding is grasping” because of our primary experience of “getting information about an object by grasping and manipulating it.” (The concept of embodiment is more involved than is appropriate to explore in this essay. The interested reader might consult the book “The Embodied Mind” by Varela, et al., 1991.) Lakoff and Johnson go on in “Philosophy in the Flesh” to explore how such embodiment shapes our understanding of the world in a way that is radically different from that taken throughout the entire history of Western logical and analytical philosophy. I suggest the idea of embodiment can also add insights to the empiricism of Purves and Lotto as a way of approaching our understanding of visual perception. For example Lakoff and Johnson note that the metaphor “seeing is touching,” as in “She picked my face out of the crowd” relates to our primary physical experience of making a “correlation between the visual and tactile exploration of objects.” Is this why sculpture in a museum has a real or implied ‘Don’t Touch’ sign at its base? When we look at the varying perspectives and shapes of cubist painting can we do so without simultaneously imaging how those shapes and edges feel to the touch? Moreover, I am suggesting we turn this desire to touch the painting to aid in understanding the artist’s way of depicting the embodied world, into clues about the processes of perception that occur in the brain. There is an expanding body of research in the field of information visualization. The challenge posed is how to take extremely large and complex information that exists in one domain, such as a huge database or the set of coordinates of the position of atoms in a biological macromolecule (e.g., hemoglobin) and represent that information in the visual domain. Are there principles that can be discerned that might guide the development of useful versus noisy and ambiguous visual representations? One approach to this question starts by trying to characterize the nature of the initial data set and the potential information in it, and then to determine what the observer wishes to know, again interpreted as information (Wehrend and Lewis, 1990). Then, an analytical Theory of Measurement (Card, et al., 1999) can be used to guide the construction of the transformation. I suggest that a complementary approach is to use the embodiment principles of Lakoff and Johnson combined with the empirical approach to seeing of Purves and Lotto as a guide to learning about and exploiting the types of transformations the brain is best tuned to accomplish. It is also necessary to examine how artists deal with information about the world by analyzing their works to add to our categories of what it is that an observer wishes to know. The transformations that an artist uses in producing art must be understood as a part of the tool set of scientific visualization.
Prof. Mark Wm. Dubin
Director for Academic Advancement,
BP Center for Visualization, UCB.
Colorado, USA