WGBH Boston: One Guest Interview

A few months back I recorded an interview with WGBH Boston that has finally come online:



They went with the "not just for kids" lines a bit, but it was a quick tantalizer into my work I thought. The most challenging thing I thought was how fast it was. We had a number of topics we wanted to hit, and only 5 minutes to do it with no editing. So, that meant everything went by very fast. Enjoy!

Comics and Film: A Narrative Perspective

Christiansen, Hans-Christian. 2000. Comics and Film: A Narrative Perspective. In Comics and Culture: Analytical and Theoretical Approaches to Comics, edited by A. Magnussen and H.-C. Christiansen. Copenhagen: Museum of Tusculanum Press.

This paper attempts to draw from film theory to inform the understanding of the comic medium. He discusses things like film shots/cuts, etc. especially in light of Bordwell and Thompson's work claiming that comprehension runs along a continuum of:

Conventions --> Norms --> Cross-cultural universals --> Deep structure of visual storytelling

For example, he claims that if knowledge of "cinematographic narration" comes from perceptual understandings, then it dispels the myth that these techniques were "invented" by early innovators of cinema. Rather, then would involve universal "deep structures of visual storytelling" grounded in perception.

Of course, this would assume that "cinematographic" techniques don't do anything that is out of the realm of perceptual knowledge, which isn't always the case — for example, with showing zooms alternating with panels of various characters ("Refiner Projection") and external settings that provide superordinate place information.

He also makes a case for complexity of storytelling matching genre, and says "as a rule, there is a higher frequency of point of view-structures in adventure comics than in romance comics" though provides no evidence for this claim.

While film uses movement, comics use static images. Thus, he breaks down aspects of temporal continuity between panels into three filmic types of cuts:

1) Matched cuts - where a movement is continued from one panel to another
2) Movement images - continuity created by action and shot-to-shot closure
3) Elliptical cuts - a discontinuous relation between shots that requires greater inference to understand the relationship

He argues that comics primarily use the final type of elliptical cuts the most, and that it is not disruptive in comics because of their layout on a page. He closes this appeal by stating that this process of ellipsis is actually a part of McCloud's notion of "amplification of simplification" for representing visual events, and that the way in which comics are understood may lie at the root of how film is understood at a base level (which I'd mostly agree with).

7 Years!

Oop, looks like I forgot to post a couple weeks ago on 5/31 that this website hit its 7th birthday! Whoohooo  — belated celebration!

Not much time to blog these days. I'm currently running an extensive experiment that has gone through numerous rounds of troubleshooting, so that's occupying most of my time. If this design works though, I'll be dancing around campus in amazement. I'm also finally writing up the results of a study that was run about two years ago, which is a nice relief.

Plus, I'm preparing my Comic-Con talk for this year that's on experiments I ran based on this blog post about "Action Stars." I'll have more up on that as the date grows nearer.

As usual though, if there's any question or issue that you'd like me to blog about, I'm always willing to take requests.

What is "Visual Language"? Video Talk

A few weeks ago I had the pleasure of giving a talk at the University of Toronto about my theories of visual language hosted by the Knowledge Media Design Institute. They have now posted this talk online to be viewed in full (beware, it's quite long) on their website.

If anyone is really interested in just what my theory entails overall, this is definitely worth watching. It lays out the basic principles and issues for what exactly I mean by "visual language", and how that relates to "comics", "language", "art", etc.

Note: Be forewarned that the slides they show do not have their full proper animations, so might end up looking a bit more cluttered than is intended.

Cartoony vs. Realistic Images in the Brain

Choi, Yeojeong, Kim, Takhwan, & Jaeseung Jeong. 2008. "EEG Source Localization during Empathy of Iconic and Realistic Cartoon Characters." Organization for Human Brain Mapping (OHBM), Melbourne, Australia ,15-19 June, 2008

In McCloud's Understanding Comics he proposed his theory of "cartoon identification" that cartoony* images are "identified" with better than realistic images. This study (pdf) tested McCloud's theory by using behavioral measures of a 7-point rating and EEG measures of the brain's electrical activity.

I've found that this theory of McCloud's was a bit ambiguous, since people have interpreted it in two different ways. It can either mean that people "identify" with cartoony images meaning...

1) They are perceived cognitively at a more "base" level.
2) That they empathize with the characters more.

Critics have usually tapped into the second reading, since it is close to a claim about how people "identify" with characters in a literary sense. However, I've always been more partial to the first interpretation (which I attempted to codify further in my book), though the two views aren't necessarily mutually exclusive. Indeed, cartoony images could evoke more empathy because they are more conceptually basic. This study aimed to examine the second version, strictly with a view of the degree of "empathy" styles create.

The experimenters contrasted comic strips that featured two opposing characters who were depicted in either realistic or cartoony styles (as in the example above) and put into different scenarios to evoke reader sympathy via who wins the confrontation (strips had varying endings, ex. winner is congratulated by a woman vs. loser is consoled by a woman). Participants only viewed one depiction, and only one option for the ending.

They then compared ratings in a behavioral study on a 7-point scale measuring empathy to both "winners" and "losers", and in a separate population, measured EEG brainwaves for the same stimuli.

Brain areas were activated that related to social perception, recognizing facial expressions, and seeing another person's pain**. They found both higher behavioral ratings of empathy and greater activation in the brain for areas for the cartoony characters than the realistic characters for both "winners" and "losers" (though different brain areas for different roles). They take these results to be support for McCloud's theory of identification that indeed, cartoony images do invoke greater empathy from a reader than realistic images.

NOTES:

*The authors, and McCloud, often use "iconic" to mean "cartoony" — I'm going to avoid this because it doesn't accurately convey what "iconic" means in a semiotic sense (i.e. meaning through resemblance). Technically, both cartoony and realistic images are "iconic."

** Just a caveat for those who actually follow the link to the pdf poster. The study shows nice pretty pictures of brains with activated regions to support its hypothesis, but these can be misleading given the actual methods used. Unlike a technique like fMRI, EEG does not give much information about where in the brain something occurs, and is much better at when it appears (i.e. the timing of processing). These brain images and results were gained using a "source localization" procedure which extrapolated from the data what brain regions were being used, a technique that is commonly employed, but often controversial. This isn't to say that the results are inaccurate, but they should be understood with this context.

Consistent reading time for comic pages?

I had a link sent to me recently asking about this blog post that claims all comic pages regardless of content are read at 3.75 seconds per page. A relevant section:

I read in one of Frederik Schodt’s excellent books on manga that a study concluded that readers spend an average of 3.75 seconds on a comic page. My own observations of myself and others has led me to believe that time frame to be fairly consistent, by which I mean not dependent on the contents of a page. Unless a writer really creates an absolutely confounding monologue or an artist completely botches an integral sequence, readers do not seem to change their flipping speed for “difficult,” wordy, nor beautiful pages. This yields somewhat counter-intuitive results, in my estimation. Single panel pages, which should ostensibly be flown through, allow one image to be lingered on or “drunk in” because that one drawing is granted the full 3.75 seconds. Pages with many panels, taken to the extreme above, should require a slower, more contemplative pace. But they do not. They seem to clock at the same 3.75, meaning the eyes need to whip through these images to make it in time.

He then goes on to advocate different strategies of layout based on the idea that readers will go through it at this magic time of 3.75 seconds. Since I wrote a lengthy counter-rebuttal to this claim, I figured I ought to post it here too.

According to the science I've seen, this does not seem to be the case. The amount of time people spend on each individual panel varies based on how much information is in it, it's order in the sequence, as well as possibly size of both panel and page, and a whole page time varies definitely the way the page layout is organized.

From a very general study of my own relating to times it takes 4 panel comic strips to be read, I found each panel at an average of 1.5 seconds per panel when readers press a button to advance through panels. But, it does vary per position and narrative structure — first panels are consistently slower, panels after major events much slower. However, if you just take that average and multiply it by 4, that gives you 6 seconds for one 4 panel Peanuts strip that has no words in it.

In my last study, I found reading times varying between .6 and 1.8 seconds per panel (small times for panels that had very little information, such as blank panels or those with just action stars), with the full 6 panel strips clocking in around 6 ±2.5 seconds.

Plus, the *uncited* study that was mentioned in the blog is for manga (and if I recall correctly, Schodt also doesn't cite the actual study), which consistently 1) use slightly less panels per page (my corpus study — "Cross Cultural Space" — showed both American and Japanese books to have 5 panels per page, but manga had a lower standard deviation), and 2) use less balloons per page. Furthermore, eye-tracking studies show that fluent readers skip over far more balloons than non-fluent readers — so, less balloons means less reading time, especially for fluent readers.

The poster here then says that he finds this time to be consistent to his own experience — but you can't know such a thing from anecdotal evidence. You would have to have measures to substantiate it.

And, even if it were true that on average pages are read at a pace of 3.75 seconds — which, I imagine there is some average time out there if one were to crunch all the numbers — there is no way that we would feel the need to allot different time to different panels based on some intuitive feeling that we "want" to read each page in a specified amount of time.

Rather, the time it takes to read a page all depends on its content and the fluency of the reader.

Congrats to my visual linguists!

I'm very excited to say a big hearty Congratulations to my research assistant Natalie, who graduated today from Tufts. She put in an incredible year's worth of work in the preparation stages of a major project of mine, apexing in the online study that many of you all took (thanks from both of us!). Last week she presented a poster of the results in a department-wide session, and did a fantastic job.



Also, I'm pleased to say that the students of my Visual Linguistics of Comics class did a great job throughout this semester, and gave me great food for thought, as well as encouragement that this is only the first of many such courses.

Public Lecture in Toronto, May 14th

For those of you in the Toronto area, I will be giving the first talk in the Visual Thinking lecture series at the University of Toronto on Thursday, May 14th at 5pm. The talk is open to the public and more information can be found here. Here's the topic of my talk:

What is "Visual Language"?: What Comics can Tell Us About the Mind

Many theories describing "visual language" have been emerging from diverse fields including computer science, communications, and design. However, often these approaches rely on metaphoric or folk notions of "language" without delving deeper into what Language actually consists of, especially on a cognitive level. This talk will present Visual Language Theory from the view of the linguistic and cognitive sciences to discuss what "language" entails, and thereby exploring just what it means to have a literal theory of a graphic modality of language. The result will be a view of graphic communication and the capacity for drawing that is embedded alongside other mental capacities and divorced from socio-cultural labels that stymie its recognition.

For those of you not in Toronto, I've been told that this presentation will be recorded and available on iTunes following the event. I'll post more on this as I get more info.

Take my Comic Experiment!

Hey everybody, I once again have another comics related experiment online and would be extremely grateful for your help!

This experiment consists of making judgements about various created comic strips, and takes roughly 20 minutes (though at first glance you may think it looks long, trust me, it doesn't take much time). Participation enters you into a drawing for a $50 Gift Certificate to Best Buy (either online or in store).

This project has a deadline of early next week (Midnight on Tues, 4/28), so your help sooner rather than later would be most appreciated.

****UPDATE: This project has been completed. Thanks very much to everyone who participated! Stay tuned for more experiments in the future, or email me to be contacted directly about future experiments.******

The study can be found online HERE.

Thanks very much!

Oh, and by the way, to all who participated in my last online study (thanks again!), a presentation of that overall project will likely be given as my talk at this year's ComicCon...

Indexing Events with Panels

A comment on my review of Magnussen's piece on semiotics in comics asked me to expand on this part:

More interestingly, she claims that the "still-images of actions" are also indexical, because they only show a part of a broader temporal whole action. This is probably the most astute observation of the piece, yet receives relatively little relative attention. This is a key insight, and would be worth expanding on.

First of all, in the semiotics of C.S. Peirce, indexicality is a means through which reference is garnered via causation or indication. For example, an index finger that points to something doesn't mean that thing, it indicates the thing has meaning. The finger is just saying "for the real meaning look over there." Also, if I saw a footprint in the sand, it indexes the person who once walked there, because of the causation stepping there created.

Another aspect to indexicality can happen through part-whole relationships. By showing just a hand, you index the rest of the body (assuming it hasn't been detached....*shudder*).

So, related to Magnussen's point, I'm recalling this particularly salient image in my mind from the book How to Draw the Marvel Way depicting a figure punching in several points throughout the overall action.

What Magnussen seems to imply is that a single snapshot of one part of this event sequence indexes the whole rest of the sequence. I would agree with this in general, though I think it's likely that different places within that progression will be more or less salient as indicating the whole.

For example, in How to Draw..., Lee and Buscema's advice is to use the maximally intense points of that sequence — the ends and beginnings of the action marked "best" or "not bad." These sections of the action seem more representative of the action than the medial parts. In semiotic terms, they would index the overall action better than the parts in the middle, which are less representative of the overall action.

Research seems to have borne out their intuition. Studies have shown that people's comprehension of events is better for the maximally preparatory and completed parts of an action, over that of the middle. In fact, even 10-11 month old infants seem to parse events through these outer boundaries.

Now, it would be unsurprising if Magnussen's statement were attempted to expand beyond the representation in single panels, out to across panels. So, let's say that two panels show both the beginning and end parts of that punch sequence. Here, there is a sense that the whole middle part of the action is indexed by seeing the ends — i.e. you know the middle happened but didn't see it.

This would be, essentially, what McCloud is arguing for with Closure. That, because we know the course of events, we "fill in" knowledge of the whole action by seeing the parts. However, even McCloud acknowledges that not all panel relations are of actions (for example, his Subject and Aspect transitions), though his notion of Closure is extended over all of them.

While I do not believe that "Closure" happens to "fill in the gutter" between panels (for numerous reasons), I do think that part of a represented action might index full actions. I don't know if I'd say that the whole action, including the middle parts, is "manifested" somehow in our minds. However, the reference of a part of an action certainly would index the concept of the whole action, by only being a sliver of it.

Review: Semiotics and comics

Magnussen, Anne. (2000). The Semiotics of C.S. Peirce as a Theoretical Framework for the Understanding of Comics. In A. Magnussen & H.-C. Christiansen (Eds.), Comics and Culture: Analytical and Theoretical Approaches to Comics. Copenhagen: Museum of Tusculanum Press. pp.193-207

Magnussen's paper attempts to apply Peircean semiotics to the comic medium, though little of the actual paper focuses on those intents. Much of the paper is concerned with the definition of "comics", which is too bad, because it is nearly irrelevant for the overall claims attempted to be made. Like others, she conflates the structural aspects of the visual medium with the definition of "comics" though, so that "comics" is a sign type built out of image parts < panel < panel sequence < comic.

She makes a parallel to Van Dijk and Kintsch's 1983 discourse approach, equating panels with propositions and citing "local coherences" between panels created through inferences (which could essentially be the same as McCloud's notion of "closure"). The "global coherence" of panels relation to the whole is interpretted through narrative schemas.

Because she conflates "comics" with stories/sequential images, she states:

"For a comic not to be a story, it should be possible to create a global coherence on the basis of something other than story-structure, and in which the local coherences are made on inferences based on parameters other than actions, actors, time and place" (198).

She then gets into the usual bind regarding whether a comic without a story is actually considered a "comic" or not. If accepting that comics ≠ sequential images, this would be of no concern.

Her main analysis focuses on the semiotic elements of the example comic. She primarily focuses on icons, indexes, and symbols, and leaves out aspects of Peirce's model regarding the nature of the "Sign vehicle" that would change many of the interpretations related to conventionality. For instance, if she had those notions, simplified icons would not be interpreted as symbolic, but just types of "legisigns".

Most of her argument though is that comics use indexical signs beyond just icons and symbols. For example, she claims that word balloons are indexical because of the attribution given by their Tails. More interestingly, she claims that the "still-images of actions" are also indexical, because they only show a part of a broader temporal whole action. This is probably the most astute observation of the piece, yet receives relatively little relative attention. This is a key insight, and would be worth expanding on.

"180º Rule"... not so much

Germeys, Filip, and d'Ydewalle Géry. 2007. The psychology of film: perceiving beyond the cut. Psychological Research 71:458-466.

This experiment tests the "180º Rule" of film editing using eye movements, and finds that no evidence for negative cognitive effect is found.

The 180º Rule claims that in film editing, when showing two characters on the left and right of a shot, it would be confusing if the next shot reversed the perspective so that the characters end up on opposite sides. Filmmakers often dance around this by using over the shoulder shots that keep characters constant to their location in the frame.

The experimenters filmed two people having a conversation sitting around a table with a constant background from all angles. They cut the conversation into 22 shots and varied the number of correct vs. reversed-angle (180º violation) shots there were. This video was then shown to participants whose eye movements were tracked, measured from the "starting point" of where their eyes were located when the previous shot ended.

The results showed that eye movements were determined almost wholly by tracking who was speaking in the frame — the agent of the shot — no matter where they were located in the frame. The results showed no evidence for confusion at 180º Rule violations, nor did it show any evidence that participants were "mentally rotating" the scene to make up for those reversed angle shots.

In other words, all claims about the ill effects of 180º violations were not confirmed. They take these findings to indicate that editing rules do not cause confusion or ruin a scene's representation, and that the content of the expression overrides the way it is represented.