Comics reading: Competence and performance

Often times when I give talks, especially concerning layout or the visual grammar of sequences of images, one of the questions inevitably says something along the lines of "But, there's no guarantee that the reader will view a page in the proper order." The high variability of possible choices or readings of a comic page makes it hard for them to accept a steadfast theory of comprehension.

However, a similar issue was at play in linguistics back in the 1950s, and was one that Noam Chomsky importantly addressed in his distinction between competence and performance. Competence refers to the (idealized) organization of rules and constraints in our minds that guides us to understand language. Performance is the vast variability that happens in real life exchanges.

For example, someone might say something like this over the phone:

I ...uh... I went *cough cough* to the store *STATIC***--oday and, like, ...um... saw *CAR HORN*--ohn from my class in the check-*hiccup*-out line.

There are lots of interruptions, unclear portions and distractions. However, most likely a listener would glean from this a sentence like:

I went to the store today and saw John from my class in the checkout line.

The rules in your head are not bothered by the messiness of the context — your attentional system can filter out a lot of it.

The same is true of reading a comic page. Let's say you start in one panel and go to another, then realize it shouldn't have gone next. You're not belying the mental rules that go into comprehension — in fact, those rules are what tell you it's the wrong order. These actual rules of comprehension are unconscious to your awareness.

Your (unconscious) competence wins out over the messiness and variability of performance.

This same issue may be at play with comparisons of comics to film. Yes, film and comics are presented differently (one static, one moving), but that doesn't necessarily mean that their comprehension in people's minds is entirely different. The difference in presentation may be a "performance" issue, while the comprehension is a "competence" issue. (Though, in my mind there is bound to be at least some variance due to that presentation difference — motion vs. static — which will need experiments to explore... yay science!).

I should point out also that, in linguistics, there are some debates over the complete reality of this split in notions. For example, for a long time it was argued that words like "um" and "uh" are just performance clutter. However, research has shown that these actually hold meaning for the discourse (essentially signaling how long a pause the speaker is going to make before continuing to talk).

Nevertheless, for many issues facing the comprehension of "comics" (and/or film), it is an important split to make.

How active is comic comprehension versus film?

Dash Shaw writes an interesting post delving into the "cinematic" nature of comics that explores thoughts from authors like Chris Ware with many insightful quotes.

Relevant to some of this discussion might be that some believe comics to have predated the film techniques. Or, the idea that this is a competence versus performance issue — that film uses the same mental structures as comics, just with a different presentation (this will be a topic of an upcoming post).

Indeed, in several experiments of mine I show comic panels one after another, one at a time where the participants have no control over the pacing. My participants have no difficulty understanding these or accepting them "as comics" (no one has ever questioned the labeling).

Most interesting though is this quote of Ware's from the post:

“I don't like to think of my work as 'cinematic.' A movie is passive -- you're watching it, taking it in. Where a comic strip, it's completely active: you have to read it, search it for meaning, for the connection with your entire experience and your memory. Yes, you do have the illusion of watching something happen in a comic strip -- but if it's done well, it comes alive on the page like a novel. A novel is the most interactive thing ever created.”

I have a lot of responses to this quote, but I'll save some for a later post. Right now, I want to question what "watching it, taking it in" means with regard to film comprehension that's different than the "active" comprehension of comics. This is a common thread in comparisions, so I wonder whether Ware (and many others who also do it) is conflating the presentation of a comic/film versus its comprehension.

Is the sense that film is "less active" because it's pace of viewing is not controlled by the viewer? This to me seems like a trivial thing in terms of comprehension. The process of understanding (i.e. piecing together the meaning between images, words, and/or sound) should maintain roughly the same.

If comprehension were different, we would expect grossly different results if we presented the same comic strips in different ways in an experiment (that could use any number of measures of comprehension). Let's say we had three different methods:

1) a comic page where all panels were laid out in a grid, possibly controlled so that subsequent panels only appear when a button is pressed by the reader ("self paced reading")
2) a "self-paced reading" task where only one panel is on a screen at a time
3) a presentation with no participant control, where only one panel appears on a screen at a time for a designated amount of time

Now, I would expect no significant difference in the ability of people to comprehend these different scenarios. This is all about presentation, not the content of the strips, since those could stay the same across all of these (and other) presentation methods.

#3 on this list is essentially the same type of presentation that film uses. Granted, I will wholeheartedly agree, film's use of *moving* images certainly does change comprehension. However, there still has to be meaningful connections between and across film shots (be it live-action or animated). These would be of the same "active" sort of connections that Ware describes. Indeed, you can replace film shots for panels in the above three options and probably get the same sort of comprehension as you would for static comics. So, instead of issues of presentation, the focus of questions should instead be on issues of comprehension, like:

How does the comprehension of static versus moving sequential images differ?

How does moving images within a unit (shot vs. panel) change its comprehension?

How does the use of moving across a scene (as in panning, zooming, etc.) differ in comprehension from it's static presentation in panels (or shots)?

AND... we can't really answer these questions without an adequate theory of how comprehension of sequential images works in the first place, which is essentially what my research for the past several years has been focused on.

Comics and Film: A Narrative Perspective

Christiansen, Hans-Christian. 2000. Comics and Film: A Narrative Perspective. In Comics and Culture: Analytical and Theoretical Approaches to Comics, edited by A. Magnussen and H.-C. Christiansen. Copenhagen: Museum of Tusculanum Press.

This paper attempts to draw from film theory to inform the understanding of the comic medium. He discusses things like film shots/cuts, etc. especially in light of Bordwell and Thompson's work claiming that comprehension runs along a continuum of:

Conventions --> Norms --> Cross-cultural universals --> Deep structure of visual storytelling

For example, he claims that if knowledge of "cinematographic narration" comes from perceptual understandings, then it dispels the myth that these techniques were "invented" by early innovators of cinema. Rather, then would involve universal "deep structures of visual storytelling" grounded in perception.

Of course, this would assume that "cinematographic" techniques don't do anything that is out of the realm of perceptual knowledge, which isn't always the case — for example, with showing zooms alternating with panels of various characters ("Refiner Projection") and external settings that provide superordinate place information.

He also makes a case for complexity of storytelling matching genre, and says "as a rule, there is a higher frequency of point of view-structures in adventure comics than in romance comics" though provides no evidence for this claim.

While film uses movement, comics use static images. Thus, he breaks down aspects of temporal continuity between panels into three filmic types of cuts:

1) Matched cuts - where a movement is continued from one panel to another
2) Movement images - continuity created by action and shot-to-shot closure
3) Elliptical cuts - a discontinuous relation between shots that requires greater inference to understand the relationship

He argues that comics primarily use the final type of elliptical cuts the most, and that it is not disruptive in comics because of their layout on a page. He closes this appeal by stating that this process of ellipsis is actually a part of McCloud's notion of "amplification of simplification" for representing visual events, and that the way in which comics are understood may lie at the root of how film is understood at a base level (which I'd mostly agree with).

"180º Rule"... not so much

Germeys, Filip, and d'Ydewalle Géry. 2007. The psychology of film: perceiving beyond the cut. Psychological Research 71:458-466.

This experiment tests the "180º Rule" of film editing using eye movements, and finds that no evidence for negative cognitive effect is found.

The 180º Rule claims that in film editing, when showing two characters on the left and right of a shot, it would be confusing if the next shot reversed the perspective so that the characters end up on opposite sides. Filmmakers often dance around this by using over the shoulder shots that keep characters constant to their location in the frame.

The experimenters filmed two people having a conversation sitting around a table with a constant background from all angles. They cut the conversation into 22 shots and varied the number of correct vs. reversed-angle (180º violation) shots there were. This video was then shown to participants whose eye movements were tracked, measured from the "starting point" of where their eyes were located when the previous shot ended.

The results showed that eye movements were determined almost wholly by tracking who was speaking in the frame — the agent of the shot — no matter where they were located in the frame. The results showed no evidence for confusion at 180º Rule violations, nor did it show any evidence that participants were "mentally rotating" the scene to make up for those reversed angle shots.

In other words, all claims about the ill effects of 180º violations were not confirmed. They take these findings to indicate that editing rules do not cause confusion or ruin a scene's representation, and that the content of the expression overrides the way it is represented.

Camera angles and meaning

Kraft, Robert N. 1987. The influence of camera angle on comprehension and retention of pictorial events. Memory and Cognition 15 (4):291-307.

Kraft explores the semantic associations made to different camera angles (high, eye-level, low) in a four frame photo story. Subjects were asked to rate the story along a 7-point scale, use a recall test for remembering the order, and then a recognition task. Each story used two characters, which were contrasted in each frame position with differing angles in different story sequence types.

Overall, results dramatically supported that angle does correlate with semantic meaning when comparing how characters were discerned. Low angles support senses of shortness, weakness, afraid, timid, and passive, while high angles were thought of as tall, strong, unafraid, bold, and agressive. A lesser correlation was found to value judgements like good/bad. Eye level angles did not contrast between characters. These results seemed to be sustained across several experimental tasks.

In recall tasks, analysis did show that camera angles influenced a connotative meaning for how characters were remembered.

Explanations for this correlation claim it comes from of our experience with the visual world, such as how looking upward at taller people gives them a sense of power (like children to adults). An alternative view says that the different angles allow the viewer to see different things in the images, from which they draw the semantic implications.

If these results extend to drawings, it would be interesting to do further study on the semantic correspondences. I find it dubious to fully believe both of the reasonings above, at least in a universal sense. There is no semantics attached to the aerial view in Australian sand narratives, nor do fixed high angles in Japanese children's representations or ancient Asian graphics have any semantic correlation that such a theory would require.

Rather, this may simply be a case of learned conventions. We've built up these meanings by continually viewing them, particularly in movies. Or... the "visual world" explanation could be valid, but only in systems that allow for flexibility in viewpoints, not those that have fixed perspective. To be honest, I had always had doubts about claims that camera angles had semantic meanings, so I'm glad there's actually work that backs it up.

Problems with Transitions

Over at Derik's blog he's been examining McCloud's panel transitions based on influence from film theory (Part 1, Part 2, Part 3, Part 4, Part 5 ...more to come).

While Derik only does it a little bit, the application of film theory to panel transitions isn't altogether new. John Barber essentially grafted McCloud's and my own (old model of) transitions onto Eisenstein's thesis/antithesis/synthesis model in his masters thesis. This was then argued against by Ben Woo in his thesis, dismissing it more because modern film theory does than any explicit argument against Barber's thesis. I'm not up on my modern film theory that much, but I believe Eisenstein is fairly passé at this point anyhow.

A few months ago I started noticing how similar Eisenstein's montage was to the cognitive linguistics notion of "Blending." Blending takes two concepts and extracts parts from them to create a new entailment. A classic example is "The surgeon was a butcher" — both surgeons and butchers are skilled at cutting flesh/meat, yet when combined together they illicit a meaning that the surgeon was sloppy. This is just like the 1+1=3 idea from montage.

And it certainly does appear across panels. I had a whole section on blending in my paper A Force of Change. Though, I think that the structures governing sequential understanding (i.e. syntax and semantics) are different from this.

Really, Eisenstein's montage and McCloud's closure are kind of like the film/comics equivalent of ether; a magical "mental" substance that doesn't really exist that glosses over any real substance the mind might actualy be contributing. They're like pop-science: a simple easy explanation for a very complex phenomenon. Just like Freud and Jung are still thought of by laypeople as being what psychology is about, their theories are far left behind to modern thinking. In fact, I'd venture to say they're more used by humanities/social sciences these days than psychology or cognitive science.

Of course, I've been railing on the panel transition approach for quite a while now, over the course of several alternative models. And, it's not just the idea of transitions that has problems: it's any approach that only takes into account panels that are immediately adjacent to each other. Any linear approach to the idea of creating meaning in sequential images will ultimately fail.

As I mentioned on one of Derik's posts, the major shift comes in what one is looking at. Instead of looking at panels' immediate surroundings and basing the system around those juxtapositions, we can instead acknowledge that whole sequences mean things (events/actions/situations/ideas). From there, it becomes a matter of identifying what functions different panels play in creating that overall meaning. Just because we read and write panels linearly doesn't mean that's how we understand them.

Nevertheless, it's interesting to watch Derik go through steps in his thinking in relation to what I did. He named it "rethinking transitions" so it'll be fun to see what his rethinking leads to.

Updated 12/1 with additional links to further entries