Perceptual Representation: Pictures and Sounds, Seeing and Hearing

Perceptions are representations. What representations are can be cashed out in several ways.

First, they are about something.

Second, they represent the world as being a certain way.

Moreover, they have satisfication notions associated with them (and corresponding satisfication conditions, i.e., conditions that must obtain for them to be satisfied). The satisfaction notion of desires is 'being fulfilled' or not; with belief it is 'being true' or not; with volitions it is 'being done' or not; with commands it is 'being obeyed' or not; and with perceptions it is 'being accurate' or not.

Representations include words, sentences, beliefs, pictures, questions, paintings, videos, signs, diagrams, maps, hand gestures, commands, recordings, desires, and, in the case at hand, perceptions.

If a given representation has a satisfaction notion associated with it, it will also follow that it has satisfaction conditions. Giving the satisfaction conditions for a type of representation is (at least part of) giving a semantics for that type of representation.

Much work has been done on giving semantics for various sorts of linguistic representations. Whole classes of sentences have been given rigorous semantics. For instance, famously, modal language has been given a whole semantic theory, rigorously formulated in terms of mathematical models involving possible worlds.

Arguably, something similar can be done for pictures.

If a rigorous semantics can be given for pictures, this may hold some promise for giving a rigorous semantics for visual perceptions. Naively, we think of vision as giving us an "image". Think for instance of Ernst Mach's drawing of his own visual field:

Mach's picture of the visual field

With appropriate modifications for the perspective and particularities of our visual field, the naive application of pictorial semantics to visual-perceptual semantics would be to straightforwardly give the same semantics for our visual perceptions as we would for a drawing of our visual field.

This is obviously simplifying hugely and leaves a host of questions unanswered. But it allows us to get a preliminary grip on things.

What I'm interested in lately is whether, suitably modified, the semantics of auditory perception will be similar to the semantics of visual perception.

You can think of visual perception as giving you a picture, where the picture includes such properties as {OBJECT, SHAPE, MOTION, COLOR, DISTANCE, SPACE, TEXTURE, CAUSE, AGENCY}. All of these properties are represented in visual perception (and, with most of them, in pictures too, at least generally).

(Note: One of the simplifications we have to make in assuming the semantics of visual perception is just like the semantics of pictures is that, as with the semantics of pictures, visual perceptions do not have a temporal-duration aspect: A good assumption for pictures maybe, which can be thought of as "instantaneous" in some way; on the other hand, a bad assumption for visual perception, which as perceptual psychology demonstrates represents motion, and an even worse assumption for auditory perception.)

So we can think of vision as giving a picture. Can we think of hearing as giving us a picture too? If we can, then if the move from pictorial semantics to visual-perceptual semantics is relatively straightforward, then so will the move be from pictorial semantics to auditory-perceptual semantics (and maybe we can even find a better representational correlate than pictures for sound; hint: recordings maybe?).

Well, clearly, hearing doesn't immediately represent all of the same properties as vision. And neither does vision represent all of the same properties as hearing.

However, that doesn't need to stop us from thinking of hearing as presenting an auditory "picture," at least if we understand "picture" as picking out a general representational structure that is in some way common to normal, physical pictures and our mental, visual images. After all, as we said before, we can think of the visual image as a picture representing such properties as {OBJECT, SHAPE, LOCATION, MOTION, CHANGE, COLOR, DISTANCE, SPACE, TEXTURE, CAUSE, AGENCY}. But then maybe we can also think of hearing as giving us an auditory representation, structurally similar to the visual image, but instead representing such features as {OBJECT, LOCATION, MOTION, CHANGE, DISTANCE, SPACE, CAUSE, VOLUME, PITCH, TIMBRE}. We might even be able to include SHAPE and AGENCY in there in some cases (almost certainly in the case of bats), though I'd have to double-check the scientific literature.

In other words, the perceptual "structure" could remain the same between visual perception and auditory perception, and indeed, some of the qualities that are represented in vision may be represented in hearing, and vice versa. For instance, just from what I've listed, the intersection of properties represented in hearing and vision will include such features as {OBJECT, LOCATION, MOTION, CHANGE, DISTANCE, SPACE, CAUSE). On the other hand, maybe it is unique to vision to include {COLOR, TEXTURE}, and maybe it is unique to hearing to include {VOLUME, PITCH, TIMBRE}.

But again, despite the differences in features that are represented in hearing and vision, the overall representational structure might be, generically, the same at a certain level of abstraction.

What would be interesting would be to work out a precise semantics for visual perceptions and a precise semantics for auditory perceptions and see where, in the details, the two actually differ. Obviously hearing and vision will have their accuracy conditions assigned by different systems of depiction. But it's exciting to think that the two might not be so far apart, that they may, structurally speaking, having a lot in common, and that light might be shed on both by looking at conventional, physical representations (such as pictures, recordings, etc.).

Ineffability: A Serious Threat to Ambitious Metaphysics

Consider a squirrel on a tree. There are things we can represent that are simply not within the capacity of the squirrel's mental structure to represent. For instance, we can represent complex mathematical truths about the shapes and physical relations obtaining in the squirrel's environment. Despite the fact that all these facts are "happening" right in his face, our little squirrel hasn't the slightest clue about them. And it's not just that he's ignorant about them like someone who doesn't know math or physics; it's that it's completely beyond a mind like his to even represent things like facts and propositions. They are completely ineffable to the squirrel.

That all seems fair enough. But here's where things start to become worrisome: Why shouldn't we think that we are in relation to some possible being in the same way that the squirrel is related to us? (Or, if you believe in God, why not think that we are related to some actual being in the same way that the squirrel is related to us?) In other words, why not think that there are some aspects of reality that are completely ineffable to us, even if maybe they are effable with respect to some other, more advanced being? Why not think that there could be some species of creature who could comprehend things that are completely beyond our mind's capacity to even represent?

This is not a new argument or anything. See for instance Ch. VI of Thomas Nagel's book 'The View From Nowhere'. Moreover, I'm sure many of us have thought of this possibility before. But do we really consider the implications of this argument? I know that I've thought of this before, but haven't drawn anything of significance from this. However, I'm realizing now that this is a very important and deep question.

Note that this view seems to go hand in hand with the idea of metaphysical realism, which holds that there is an objectively existing world that is in some sense totally (or at least mostly) mind-independent. If you hold that view, it seems rather strange to think that somehow all of reality, of necessity, must be in principle comprehensible to us. And if reality were in its totality to be comprehensible to us it would be a rather strange coincidence. Moreover, there seems to be nothing particularly special about us. We are on a continumm with non-sentient creatures, insects, and squirrels on one side and with angels and God on the other (and probably a lot of things in between). Hence, it seems quite likely, on realism, that we are in a situation similar to that of the squirrel: There are aspects of reality which are simply beyond the representational capacities of our minds.

But then there is trouble. Interestingly, those who have high hopes for metaphysics tend to be metaphysical realists, but that very same metaphysical realism tends to undermine the high hopes for metaphysics. For example, suppose we characterize metaphysics as the study of the most fundamental or general aspects of reality. Suppose moreover that we are metaphysical realists. Then, probably, there are aspects of reality which are simply beyond the representational capacities of our minds. But in that case, for all we know, the most fundamental or general aspects of reality are within the sphere of things that are completely ineffable to us. So there is reason to doubt the possibility of having any substantial metaphysical knowledge.

In fact, maybe by a similar though distinct argument we can get a stronger conclusion. We have some reason to think that as minds become more advanced on the "great chain of being" that I've described, they become able to represent more (and more) fundamental aspects of reality than those before them. Higher beings have concepts that are more fundamental than those of the minds on lower levels. (Technically, they have concepts of things that are more fundamental.) And they probably have more of them. For instance, some lower animals can probably represent things like 'cause' and 'object' and even 'agent', but it seems doubtful whether bees could do the same (or to the same degree). But in that case, granted we are probably pretty far from the high end of the continuum, probably the most fundamental aspects of reality are only representable by beings on the higher end. So, probably, we cannot represent the most fundamental aspects of reality. So, probably, ambitious metaphysics is hopeless.

This might seem like a fun philosophical puzzle, but actually it is rather important, because if I sit down and ask myself whether I really think metaphysical realism is true, I am with utter and literal sincerity inclined to say, "Yes." And if I sit down and ask myself whether I really think some aspects of reality are ineffable for the reasons described, I am with utter and literal sincerity inclined to say, "Yes." And, to bring the trilemma to completion, I have high hopes for metaphysics and sincerely think it is essential to truly understanding the world.

What to do then? Does the argument against substantial metaphysics work? What are the implications for metaphysics and other areas of philosophy depending on which way one goes? How might different views solve the issues here?  These are interesting questions. Since I've been thinking about this stuff for a class I'm taking, I'll probably have a chance to write a term paper on it. I have inklings about where we might go, but I have no clear answer at the moment.