This blog has moved

This blog is now located at http://blog.emaki.net/.
You will be automatically redirected in 30 seconds, or you may click here.

For feed subscribers, please update your feed subscriptions to
http://blog.emaki.net/feeds/posts/default.

Metaphors go boom!

Derek Kirk Kim uses a fun and common conceptual metaphor in today's installment of his ongoing series "Praxis & Allies" (which you should be reading). Here, the character's head explodes on discovering his crush actually likes him.

"Conceptual metaphor" is a notion that's been prevalent in some circles of linguistics for the last 30 years, and is a phenomena where one domain of ideas is mapped onto another. The metaphor that Derek uses is "Emotion is Hot Fluid in a Container" — the emotion constitutes "pressure" in the head (the container) that then can "erupt" when it "overflows." Emotion isn't actually hot fluid in a container, but we map the idea of emotions onto the domain of a container.

There have actually been several papers written on this topic with regard to comics, usually describing Anger. The common visual sign of smoke coming out of an angry character's ears directly links into this "Anger is Hot Fluid in a Container" metaphor. But, here Derek uses it just for an emotional overload. A similar usage was done years ago in Journal Comic by Drew Weing.



As has been argued by many, using these sort of cross-domain metaphors is a great way for the graphic form to visually portray things (like emotions) that aren't otherwise visible.

Defense!

I'm very excited to say that, after working on this project for 2.5 years, I'll finally be defending my Master's project — "Balancing Grammar and Semantics in 'Comics': Global Structure in Sequential Image Processing" this coming Monday April 12th here at Tufts. The presentation will describe two experiments that together show converging evidence that the comprehension of sequential images — as in comics — uses a grammar, similar to the way that sequential words use a grammar. Here is my abstract in Haiku:

Image sequences
Grammar, Meaning — separate?
RTs, ERPs.

It is a public defense, so if you actually want to come, you're more than welcome to email me for more info. A shortened version of this presentation will be what my talk is about at Comic-Con this year...

Updates

So, I unfortunately haven't had much an opportunity to blog lately. My apologies to the usual readers! Some updates...

I have been working hard at finishing my latest projects, which I had the fun opportunity to present as a poster at the CUNY sentence processing conference out here in New York City. To any readers that found this blog through the conference, thanks for checking out my poster and the site!

I'm currently working on writing up those experiments, which I'm very excited about. I also have a few publications in the works, which I'll be updating about once they're due to make an appearance. Plus, with my assistants, we've have been working on some projects that should have some exciting results soon.

I do have some backlogs of intended blog topics, so hopefully I'll be back to posting more entries soon...

Action Stars and Smoke-veiled fights

I've posted every now and again about a convention in comics that I've called "action stars", where a whole panel is replaced by a star shaped "flash" that essentially represents "event happens here!" but doesn't show that event. I've likened this to being like a pronoun in the visual grammar, since it can replace the Peak events of the sequence, just like a pronoun can replace a noun (or noun phrase).

Over the past year I've run some successful experiments using action stars, and am planning a few more of them. But, I've also had the lingering question whether there are any more of these "visual pronouns" out there...

And I think I've found one.

Another common piece of visual morphology is the "smoke-veiled fight" (alternative names welcomed), where a big puff of smoke is shown with arms and legs sticking out, to stand for a fight occurring, which can also take up a whole panel:



Some interesting contrasts can be made between the smoke-veiled fight (SVF) and action stars. First off, SVF panels are much more restricted; they can only appear for fights, whereas action stars can go on almost any Peak panel. We might write this out this difference in meaning formally as:

Action Stars: [Event: X]
SVF: [Event: FIGHT(A,B,...n)]

This basically says that an action star carries the unspecified meaning that an Event "X" occurs, but SVF panels show an Event of "Fighting", consisting of at least characters A and B up to "n".

Also of interest is that while both depict "events", the nature of those events is intrinsically different. Action stars show a single event, while SVF panels show a duration. Notice, you can't glean the sense of duration from an action star, nor can you interpret the SVF as a single event. But, the difference is there — even though in neither one can you actually see what events are "actually" happening!

"Defining Comics" video

Patric Lewandowski offers this video lecture of his attempt to define comics, based on his earlier column from Comixtalk. He covers a lot of ground, meshing numerous memes of comics theory.

Ultimately, I do disagree with most of his points, for reasons I've described elsewhere**, but it's at least interesting to see him present it all together, and I do like that other people are at least trying to address these issues.

While I've defended people's attempts to define comics before, from the visual language perspective, the whole issue of "defining comics" does seems a little strange, and likely stems from McCloud's big thrust to do so in Understanding Comics.

McCloud's guiding rhetoric was a division of form and content. For him, the form was comics and the content was the genres that appear in comics. But, you can take this one further, since comics are only a "form" if you presume them to be.

Really, comics are made of two mediums: text and sequential images. These can be the "form" and the notion of "comics" is the content. Truly, as Patric tries to get at, text and/or images appear in lots of places, but only sometimes are they called "comics." This says to me that "comics" are not a thing definable by those elements at all as a type of "form."

Rather, comics are written in text and images the same way that novels or magazines are written in text. From this perspective, debates over "what are comics?" are rendered similarly to "what are novels?" or "what are magazines?"


-----
**Though I will add emphatically that my theory of visual language as he presents it there is fairly misleading. Visual language is NOT just about iconography!!

5 Card Nancy and Panel Transitions

One of Scott McCloud's more wacky inventions is the game Five Card Nancy which is based on the old comic strip Nancy. The basic premise of the game is that you can create lots of different (and fun) novel strips by combining random panels together. Scott recently posted an old collage he did that led to the game.

Of immediate note in his collage is that the sequence doesn't exactly make much sense, despite some cohesion between the panels. I'd say that it may have a narrative structure (i.e. visual grammar), but no meaning (semantics).

In some cases though, the juxtaposed panels do make sense, but the global meaning does not. In linguistics (borrowed from math), we'd call this a "first-order Markov chain", since only the units right next to each other have a connection. If a panel had a connection to two panels next to it, it'd become a "second-order chain", etc...

Markov chains were the primary way that people thought about language's grammar up until the 1950s, when Noam Chomsky then showed that grammar needed to account for connections farther than just countable individual word relationships (an approach I then applied to comics' sequences).

Essentially, McCloud's theory of panel relationships is a first-order Markov chain theory. It only looks at juxtaposed relationships. Interestingly, his Five Card Nancy game follows the same characteristic. Since players put down one panel at a time, it appears as though they are just making choices linearly. However, I'm guessing that the higher scoring combos are all ones that gel on a global scale, not just a local connection.

Also, the limitation of the panel transition viewpoint is really highlighted by McCloud's Nancy collage. How can panel transitions be correct if only local connections make sense but ones further down the sequence do not? Though we may draw and read comics one panel at time, it doesn't mean we don't build or project a bigger structure in our minds beyond the linear relations.

Storycards and visual grammar

My friend Alex sends along this link to a gift pack of "storycards". Basically, you can use these cards in sequences to create lots of different novel stories. The idea is similar to McCloud's Five Card Nancy game.

I'm interested in it for a few theoretical reasons. For example, having a stock set of units that can be combined in different ways is similar to language, where you have a set of words (vocabulary) that are combined in various ways (grammar). As the main thrust of linguistics in the last 50 years has told us, infinite possible sentences can be made with just a small set of vocabulary items, and that's basically the fun of such a card set!

However, even more interesting, its very similar to a study I finished running a few years ago and am still working on getting written up. In it, my participants were given four panels from a Peanuts strip and asked to arrange them in an order that makes sense.

People were very good at getting the original order of the strip (around 90% if I remember correctly), though that's not what I was interested in. I was more interested in the errors that people made, and whether there were patterns to them. And, indeed, there were. Prior to testing, we had coded the panels for numerous narrative properties, and found that certain narrative categories got moved around in particular patterned ways.

What this showed was that people don't just make up sequences one panel at a time as this game suggests, but that elements of that order are conditioned by roles of panels. These roles are determined both by properties of individual panels, and the relations between images.

So, one-by-one reading/drawing, but guided by underlying complexity (grammar) beyond just linear relationships.

Almost back...

Okay, one more status update then we're back to some real posts...

My brainwave study has nicely concluded and I've now moved on to analyzing the results. Things look fantastic, so I'm very excited about working to get these written up and submitted to a journal in the coming months. This whole project has been a very long one: 1.5 years making stimuli, 7 months running the experiments, probably another 3 or so writing it up. It'll be a relief once its done (and the next beginning), but the results seem to be worth it...

Otherwise, I've finally had some time to tinker with some decent blog posts. So, I'm just going to let them accrue until after the holiday weekend then come back and start posting again. It's been a busy busy few months, but I think some more blogging will be back in the mix shortly. Phew!

Not MIA!

Unfortunately blogging has had to take somewhat of a back seat lately, as my workload has been just crazy.

At the end of this week I should finish running participants through my brainwave study using comics, though I can already confirm that the results are just fantastic. I think I can confidently characterize how the brain processes narrative sequential images, and its really quite exciting (with pretty solid evidence against panel transitions). Stay tuned... (though I'll say already that this will be the topic of my ComicCon talk in summer).

Otherwise, I've been working on writing up several other studies that have been ruminating in my computer. What with the multiple articles currently under review for journals and books, over the next few years I should have a steady stream of new papers emerging.

Beyond that, the publisher De Gruyter has been nice enough to provide me with a review copy of the recent volume Multimodal Metaphor which has lots of chapters on comics. Once I have a little more time I'll be writing reviews of the book and relevant chapters.

Transition Overload!

I've frequently heard it said that every panel in a comic has to connect to every other panel. I've tried to go about showing the problems with individual transitions or McCloud's closure, but I have yet to tap into this issue on the blog.

Potentially, this could be at least somewhat the notion behind Groensteen's ideas of braiding and arthrology. "Restrained arthrology" says there are meaningful connections made between all juxtaposed panel relationships (i.e. what McCloud would call panel transitions), while "general arthrology" pushes this up to possible connections between all panels in a book ("braiding").

In my book, I toyed with a similar idea of multi-connected transitions for very specific examples, but cast it aside before proposing my alternative approach based on Chomsky's generative grammar. However, the "every panel with every other" viewpoints are far more unconstrained than my approach ever was.

One of the biggest problems with this "every panel with every other" as a theory of comprehension is that it would just overwhelm a person's working memory to keep that many things active in their head with no guiding structures. So, I figured it would be worth the exercise of showing how ridiculous such an assertion might be...

For an average book that has 6 panels per page for 24 pages, this would give 144 panels in a book. Connections between any two panels in those 144 would be calculable as 144!/(2!•142!). This would build up to 10,296 possible transitions as every possible combination would additively create with each successive panel read, as the mind continuously retained them all in memory. Granted, not all panel relationships might need to establish an explicit "transition", but all connections would be necessary to at least confirm or deny the need for an explicit transition.

Without any underlying structure to guide such connections, this would be overwhelming for human memory to handle. Rather, there needs to be something explicit provided by the mind to manage (and group/subdivide) such connections— just like a grammar for language. Transitions and general principles of "arthrology" just won't do it.

Learning to read your brain(waves)

So, today marks a minor milestone for me, as I ran my very first study of comics looking at people's brainwaves. The image here to the right is from that first participant, and each of the lines is of a different type of sequence that we are experimentally testing.

So, what does this tell us?

Absolutely nothing.

Yet.

Data from one participant doesn't say much, but give me a few more weeks and these waves will be (hopefully) showing interesting information about how the brain processes sequences of images.