Transition Overload!

I've frequently heard it said that every panel in a comic has to connect to every other panel. I've tried to go about showing the problems with individual transitions or McCloud's closure, but I have yet to tap into this issue on the blog.

Potentially, this could be at least somewhat the notion behind Groensteen's ideas of braiding and arthrology. "Restrained arthrology" says there are meaningful connections made between all juxtaposed panel relationships (i.e. what McCloud would call panel transitions), while "general arthrology" pushes this up to possible connections between all panels in a book ("braiding").

In my book, I toyed with a similar idea of multi-connected transitions for very specific examples, but cast it aside before proposing my alternative approach based on Chomsky's generative grammar. However, the "every panel with every other" viewpoints are far more unconstrained than my approach ever was.

One of the biggest problems with this "every panel with every other" as a theory of comprehension is that it would just overwhelm a person's working memory to keep that many things active in their head with no guiding structures. So, I figured it would be worth the exercise of showing how ridiculous such an assertion might be...

For an average book that has 6 panels per page for 24 pages, this would give 144 panels in a book. Connections between any two panels in those 144 would be calculable as 144!/(2!•142!). This would build up to 10,296 possible transitions as every possible combination would additively create with each successive panel read, as the mind continuously retained them all in memory. Granted, not all panel relationships might need to establish an explicit "transition", but all connections would be necessary to at least confirm or deny the need for an explicit transition.

Without any underlying structure to guide such connections, this would be overwhelming for human memory to handle. Rather, there needs to be something explicit provided by the mind to manage (and group/subdivide) such connections— just like a grammar for language. Transitions and general principles of "arthrology" just won't do it.

Indexing Events with Panels

A comment on my review of Magnussen's piece on semiotics in comics asked me to expand on this part:

More interestingly, she claims that the "still-images of actions" are also indexical, because they only show a part of a broader temporal whole action. This is probably the most astute observation of the piece, yet receives relatively little relative attention. This is a key insight, and would be worth expanding on.

First of all, in the semiotics of C.S. Peirce, indexicality is a means through which reference is garnered via causation or indication. For example, an index finger that points to something doesn't mean that thing, it indicates the thing has meaning. The finger is just saying "for the real meaning look over there." Also, if I saw a footprint in the sand, it indexes the person who once walked there, because of the causation stepping there created.

Another aspect to indexicality can happen through part-whole relationships. By showing just a hand, you index the rest of the body (assuming it hasn't been detached....*shudder*).

So, related to Magnussen's point, I'm recalling this particularly salient image in my mind from the book How to Draw the Marvel Way depicting a figure punching in several points throughout the overall action.

What Magnussen seems to imply is that a single snapshot of one part of this event sequence indexes the whole rest of the sequence. I would agree with this in general, though I think it's likely that different places within that progression will be more or less salient as indicating the whole.

For example, in How to Draw..., Lee and Buscema's advice is to use the maximally intense points of that sequence — the ends and beginnings of the action marked "best" or "not bad." These sections of the action seem more representative of the action than the medial parts. In semiotic terms, they would index the overall action better than the parts in the middle, which are less representative of the overall action.

Research seems to have borne out their intuition. Studies have shown that people's comprehension of events is better for the maximally preparatory and completed parts of an action, over that of the middle. In fact, even 10-11 month old infants seem to parse events through these outer boundaries.

Now, it would be unsurprising if Magnussen's statement were attempted to expand beyond the representation in single panels, out to across panels. So, let's say that two panels show both the beginning and end parts of that punch sequence. Here, there is a sense that the whole middle part of the action is indexed by seeing the ends — i.e. you know the middle happened but didn't see it.

This would be, essentially, what McCloud is arguing for with Closure. That, because we know the course of events, we "fill in" knowledge of the whole action by seeing the parts. However, even McCloud acknowledges that not all panel relations are of actions (for example, his Subject and Aspect transitions), though his notion of Closure is extended over all of them.

While I do not believe that "Closure" happens to "fill in the gutter" between panels (for numerous reasons), I do think that part of a represented action might index full actions. I don't know if I'd say that the whole action, including the middle parts, is "manifested" somehow in our minds. However, the reference of a part of an action certainly would index the concept of the whole action, by only being a sliver of it.

Closure's assumptions

Patric continues his defining of "comics" with a discussion of "closure." I've talked before about the problems with the idea of closure, but it strikes me that there are a few underlying issues that people run into when addressing these issues:

1. They assume that time passes between panels, despite there being no evidence that each panel represents a "moment in time." With this assumption in place, it forces people to assume that some "moment" also lies between the panels, when no hidden moment may exist. I wrote my essay "Time Frames...Or Not" about various reasons why this assumption isn't true.

Even McCloud bungles this. While in one place he tries to say that "panels=moments" because "time=space", in his own transitions he includes three that have nothing to do with time at all! (Subject, Aspect, and Non-Sequitur transitions). For the adamant, what are the moments and what are the transitions in this "comic"?

2. People are just looking at the relationships of two juxtaposed panels. Most stabs at sequential meaning, like Patric's or Derik's, have just talked about two-panel pairs. But, rarely are sequences confined to two panels.

Just because we experience reading sequences of images linearly doesn't mean that is how we understand them. In most cases, we can easily acknowledge that whole sequences mean something beyond just paired panels. Looking beyond the scope of immediate panel relations quickly forces a rethinking of the accuracy of a view about closure/transitions.

Here are a few illustrative exercises that people can do to think more about these issues (and are things I did when first getting into this seriously):

1. Actually try to catalogue the "transitions" in a full comic á la McCloud's counting. Note any problems in the categories and where descriptions become more difficult.

2. Take comic pages/strips and sketch out the different relationships of every panel to each other. Which panels need connections, which don't? What do the relationships tell you?

If anyone actually does this, I'd love to hear about their results. In the meantime, if people are curious about my alternatives to closure, I recommend watching this video.

Coercion... of meaning!

Today I gave my big first year project presentation to the psychology department. From what everyone has said, it went very well. Of course, the project itself is still underway, and I will be running several more subjects in the lab, while probably continuing the study as a whole throught the summer. This shot is of me and my advisors from afterwards. (R to L: Ray Jackendoff, Phil Holcomb, Me, Gina Kuperberg):

As I've mentioned before, my talk involved looking at the "Event Related Potentials" or ERPs involved with processing a certain type of linguistic phenomenon called "semantic coercion." ERPs are a measure of the electrical activity of the brain. We don't get a good fix on specific brain areas that are at work, like in fMRI, but we do get very detailed analysis of the time course of events and certain waveforms do seem to indicate types of brain functioning in contrast to each other. We measure this electrical activity by sticking a cap of electrodes on people and feeding the signals into a computer, which then averages out the noise over several subjects and trials to give a smooth wave for time locked events. Here's me in the cap...

So, I looked at these brain waves for semantic coercion, which involves the extraction of "hidden" meaning from sentences like The chef finished the chicken before the main course. Someone can't literally "finish a chicken," they have to finish doing some action with it, like cooking. Since the event isn't stated outright, it's said to be "coerced" from the combination of the verb "finished" and the direct object "the chicken." Here's a waveform from one of the sites on that cap that I got in the experiment:

While this is interesting as a linguistic phenomenon, I think it's really just a warm up for more comic related studies. Since I couldn't resist, I even opened my talk by showing this strip:

Now, if you look carefully, coercion happens here too. We never see the event of Snoopy catching the ball, yet we know the event happens based on the information provided by the other panels. In addition to other things, coercion is perhaps one of the things that McCloud was trying to get at with his notion of "closure." In many ways, coercion here is an invisible meaning that is created out of the visible components of the graphic sequence. Graphically, it's the stuff that happens "out of view" of the panels. The problem is that McCloud extended this to the (linear) relationships between all image sequences, which just doesn't work.

So, if I do find anything fairly robust in the ERPs for verbal coercion, perhaps a study of visual language coercion could be on the horizon as well? Or perhaps a theoretical paper first...

Time essay analyzed

In Derik's continuing exploration of panel transitions (Part 1, Part 2, Part 3), he does an interesting job of dissecting my latest essay "Time Frames... Or Not." To keep things localized, I'll make my responses there, but he seems to have done a fairly thourough job of it. Worth perusing.

Also, Blogger has helpfully decided to add tags in finally, so I'll be trying to work those into all my past posts in due time.

Problems with Transitions

Over at Derik's blog he's been examining McCloud's panel transitions based on influence from film theory (Part 1, Part 2, Part 3, Part 4, Part 5 ...more to come).

While Derik only does it a little bit, the application of film theory to panel transitions isn't altogether new. John Barber essentially grafted McCloud's and my own (old model of) transitions onto Eisenstein's thesis/antithesis/synthesis model in his masters thesis. This was then argued against by Ben Woo in his thesis, dismissing it more because modern film theory does than any explicit argument against Barber's thesis. I'm not up on my modern film theory that much, but I believe Eisenstein is fairly passé at this point anyhow.

A few months ago I started noticing how similar Eisenstein's montage was to the cognitive linguistics notion of "Blending." Blending takes two concepts and extracts parts from them to create a new entailment. A classic example is "The surgeon was a butcher" — both surgeons and butchers are skilled at cutting flesh/meat, yet when combined together they illicit a meaning that the surgeon was sloppy. This is just like the 1+1=3 idea from montage.

And it certainly does appear across panels. I had a whole section on blending in my paper A Force of Change. Though, I think that the structures governing sequential understanding (i.e. syntax and semantics) are different from this.

Really, Eisenstein's montage and McCloud's closure are kind of like the film/comics equivalent of ether; a magical "mental" substance that doesn't really exist that glosses over any real substance the mind might actualy be contributing. They're like pop-science: a simple easy explanation for a very complex phenomenon. Just like Freud and Jung are still thought of by laypeople as being what psychology is about, their theories are far left behind to modern thinking. In fact, I'd venture to say they're more used by humanities/social sciences these days than psychology or cognitive science.

Of course, I've been railing on the panel transition approach for quite a while now, over the course of several alternative models. And, it's not just the idea of transitions that has problems: it's any approach that only takes into account panels that are immediately adjacent to each other. Any linear approach to the idea of creating meaning in sequential images will ultimately fail.

As I mentioned on one of Derik's posts, the major shift comes in what one is looking at. Instead of looking at panels' immediate surroundings and basing the system around those juxtapositions, we can instead acknowledge that whole sequences mean things (events/actions/situations/ideas). From there, it becomes a matter of identifying what functions different panels play in creating that overall meaning. Just because we read and write panels linearly doesn't mean that's how we understand them.

Nevertheless, it's interesting to watch Derik go through steps in his thinking in relation to what I did. He named it "rethinking transitions" so it'll be fun to see what his rethinking leads to.

Updated 12/1 with additional links to further entries

New Essay: Time Frames... Or Not

Wow, its been a really long time since I last posted a new downloadable essay. Well, if you've been anxiously awaiting one, today is your lucky day! I've just posted my latest theoretical offering, "Time Frames... Or Not," where I tackle the assumptions that lead to the (false) belief that successive panels equal moments in time. Here's the full abstract:

The juxtaposition of two images often produces the illusory sense of time passing, as found in the visual language used in modern comic books. While this linear sequence may seem on the surface to present a succession of individual moments, the understanding of graphic narrative is hardly so simple. This paper will explore how the linearity of reading panels and the iconicity of images create various assumptions about the conveyance of meaning across sequential images relation to space and time.

Astute and long-time readers of this blog will remember that I mentioned writing this paper way back in January of this year. Its good to finally get it done and out!

Like many, I'll be back to school come Tuesday, so hope you all enjoy the day off!

Problems with Closure, part 4

In my last post, I pointed out the assumption that pictures are not connected to any mental apparatus. I now continue on to show how that affects analysis of sequential images…

Assumption #3: Absence of Mind

By minimizing the contribution of the mind, a simple theory like closure can easily emerge. The images’ meanings are “out there in the world,” so all the mind needs to contribute is possible ways to pull those meanings together. Since no mind is found in the actual images, its placed instead between the images. Transitions just become a surface grafted onto this encompassing unifying process, where the “mind” “fills in the gaps.”

But, what is it "filling in the gaps" with? It must carry some information in order to do this.

Of course, the non-mental explanation says that we understand closure because we’ve had experiences in life that allow us to combine events in images. True enough. This is an appeal to the things being referenced. However, it still can’t escape the mental part of receiving those experiences and drawing upon them to understand images (i.e. doesn’t the mind then have to do something in order to make those experiences understood?).

This view casts the mind as a “magic box.” Stuff goes in, a conscious understanding is reached, but how did it do it? Cognition! Ok, yes, that’s true, but now tell me what that cognition is and how it works. You can’t just say “the mind does it” – you need to say what the mind does to be able to say that “it” does anything. Otherwise you’re just making an empty statement.

Closure doesn’t really say anything about the content of the panels, saying that meaning is created in the space between them. It cedes out a non-role to the “mind,” thereby passing the buck of meaning making to the ether. This makes closure essentially a faux cognitive process. And this is also why it can be extended to apply to just about anything at all.

Instead of a non-principle like “closure,” we can lay out mental schemas for events (and more) in our minds that allow for understanding sequential panels. Rather than a generalized magic that the “mystical mind” performs, this actually identifies the contribution of the mind.

My first model had three of these:

1) Environmental Phrase: unified various environmental elements at the same state
2) View Phrase: combined the same element at the same state
3) Temporal Phrase: unified elements of state changes

These "phrase structures" could then embed into each other, forming a hierarchy showing exactly what the mind brings to the table. While the panels are linear, the structures of understanding are not. Note also, by formulating these rules, they inherently pose constraints to which sequences come out.

My newer approach builds off of this further to stipulate actual grammatical roles, while rejecting the schemas above (because they don’t work entirely). You can see a glimpse of this new approach in the essay "Initial Refiner Projection", though that’s only a small part of it.

In all of these, a contribution of the mind is identified. It is not magically glossed over, and it imbues the power of meaning making to the images themselves in concert with given mental rules.

Once you come to this conclusion though, it raises some other important questions:

Where do these mental schemas come from? (learning or genetics?)
How many are there, and how do they work?
Do these structures connect to other mental domains?

All of these are very important questions, and just the sort of thing that will hopefully occupy a good deal of time and effort in cognitive science in the years to come.

Problems with Closure: Part 1, Part 2, Part 3

Problems with Closure, part 3

In my last post I pointed out that pictures are not believed to have constraints on them, and that the mind must place constraints on any sort of understanding:

Assumption #2: The Veil of Iconicity

This assumption is that pictures are “out there in the world,” not learned information, and thus not mental phenomena. McCloud shows this underlying belief by stating:

“Pictures are received information. We need no formal education to “get the message.” The message is instantaneous. Writing is perceived information. It takes time and specialized knowledge to decode the abstract symbols of language.” (p. 49)

This belief is formed because images are most often iconic, meaning that they derive their meaning through resemblance to what they reference. A picture of a person is known to refer to a person because we know what people look like in the world. Note, there are three parts to this equation: the picture of a person, people in the world, and the concept of people in our minds.

However, just because they look like what they mean, it doesn’t mean that pictures aren’t conceptual information. Through this resemblance, we forget that it actually requires a mind to understand these images, and thereby discount its contribution to understanding. Images just seem like what we experience in the world: we don’t seem to need any special understanding to know the world, so thus we don’t need special understanding to know images.

Upon closer reflection, this is somewhat of a ridiculous mistake. If I draw a picture, how can it not be connected to my mental understandings? It came out from my mind, why wouldn’t its reception need to go through my mind too!? I had to learn how to draw, doesn’t that mean I had to learn how to understand drawings too!?

Considerable studies have shown that the understanding of images is clearly not so transparent. Often, this is found in native communities like Australian or Amazon aborigines who couldn’t/can’t understand aspects of "Western” representation. In the past, this was haughtily used to justify their intelligence as "primitive" compared to Ameri-Europeans. Really, this is just a case of not having fluency in the conventionality of a graphic system (natives for the Western system(s), and Westerners for the native systems). Science is rife with these sorts of examples treating the world “objectively” while really being unable to see beyond the petri dish that oneself is standing in.

Because images look like what they represent, we gloss over the mental component for understanding them, and in turn is misplaced for sequential images. I’ll take this up in my next post.

Problems with Closure: Part 1, Part 2, Part 4

Problems with Closure, part 2

The first problem with closure isn’t a direct one, but a tendency of the way our culture treats graphic images as a whole. As a topic, it also happens to nicely correspond to my new article up at comixpedia:

Assumption #1: Artistic Freedom

In line with an “Art” perspective, there is a tacit assumption that “anything goes” with regards to graphic creations. Because “Art” is supposed to be about innovation and interpretation, anything visual is regarded as free from constraints of any sort. This is why transitional approaches like McCloud’s allow for a “non-sequitur” transition, because it’s a catch-all for any panel-to-panel relationship that might seem odd.

Closure, as an idea, allows for this sort of “anything goes” freedom, because it only involves one-to-one panel relationships. Since only two panels are looked at, it escapes the types of constraints posed by an approach that focuses on the relationship of multiple panels to each other.

Of course, if our minds are involved at all, then there must be constraints. How could the mind function without them! Even given the Art perspective, constraints aren’t easy to find anyways because people tend not to find them unless they are broken. And if constraints exist to make people make sense, it usually means that they aren’t broken all too often in daily use.

This is also a concern about the difference between the understanding and interpretation positions that I mentioned in my last post. As cognitively wrong something might seem at a base level of understanding, we can still consciously give explanations for how it might make sense under the right "interpretation." Again, the trouble comes from thinking these are the same thing.

All this concern about “mind” leads to the next underlying assumption though, which will be discussed in my next post…

Problems with Closure: Part 1, Part 3, Part 4

Problems with Closure, part 1

I’ve been reading Mike Meginnis’ blog lately, which has stirred up some observations I have about the issues of closure. As astute readers of my work know, I don’t buy into the ideas of closure anymore. I did once — developing more panel transitions even — but not since four years ago when I realized that it couldn’t possibly work for fairly straightforward empirical reasons.

Closure as a psychological notion from the Gestalts is largely about “image constancy,” which means that you can have a single image with pieces missing and still understand the whole. In daily perception, we experience this anytime one object covers up parts of another one. Despite that part is covered up, we still understand that there is a whole object beneath it.

McCloud extends closure to do this unification across two separate images. It is a compelling entailment to believe that across two panels we merely are “filling in the blanks” for events rather than objects. Or, even, extending this into some philosophical sense that we “fill in the blanks” all the time in daily life for everything. It is, however, wrong. And it’s founded on some basic underlying assumptions that I will articulate over the next several posts.

I will say, though, that I think linear transitions are the intuitive place to start an analysis of sequential images, and McCloud gave a gift to us all by inaugurating this field. But as much as I love the guy and appreciate his contributions, I have to tear into the ideas…

On the surface, closure provides what every body wants out of a theory. It’s a simple, catch –all that imbues the “reader” with individualistic power of contribution to the piece. However, human biology and cognition are rarely simple – nor should they be — given the millions of years of evolution and development they have undergone to reach the point they’re at.

Also, because of how simple McCloud leaves it, he opens the door for it to be applied to various purposes:

1) Is this “filling in the gaps” about unconscious understanding?

2) Is it about conscious interpretation of an “artistic” intent?

These distinctions are very important, and they are just what Meginnis struggles with. On one hand, we’re talking about mental processes that underlie understanding in a very basic sense. The other position is talking consciously interpreting "meaning" beyond that fundamental level of understanding.

Here’s the difference in an analogy to spoken grammar: No matter how much literary theory can explain what the "meaning" of a sentence is, it still doesn’t go into any part of real understanding. While it may debate the senses of words, the author’s intentions, and how well they achieve them, etc., it never actually broaches how the words themselves are strung together in a meaningful whole structurally.

You can debate all you want about what the "meaning" of the last sentence was interpretively, but none of that can go to explaing just why your mind can directly connect the word "debate" to each of the groupings of words "senses of words," "the author's intentions," and "how well they acheive them" (or how those groupings of words are connected to each other). Those understandings certainly aren't linear, which is how you just experienced them consciously in reading.

Such structural concerns are left to linguists, and are largely irrelevant to these “interpretive” questions because they are at a level above the structural investigations of cognition. The same is true for sequences of images.

In the next several posts, I’ll be going more in depth on the problems with closure. These will all be based on underlying problems with the theory, not delving into the empirical examples found in data which invalidate linear analyses (of which there are many).

Problems with Closure: Part 2, Part 3, Part 4