Learning to read your brain(waves)

So, today marks a minor milestone for me, as I ran my very first study of comics looking at people's brainwaves. The image here to the right is from that first participant, and each of the lines is of a different type of sequence that we are experimentally testing.

So, what does this tell us?

Absolutely nothing.

Yet.

Data from one participant doesn't say much, but give me a few more weeks and these waves will be (hopefully) showing interesting information about how the brain processes sequences of images.

(^_^) ... Emoticons and the Brain

Masahide Yuasa, Keiichi Saito,Naoki Mukawa. 2006. Emoticons convey emotions without cognition of faces: an fMRI study. Conference on Human Factors in Computing Systems. April 22-27, 2006. Montréal, Québec, Canada

Related to the previous post on a testing of McCloud's "Cartoon Identification Theory" on cartoony vs. realistic images in the brain, here's a study using fMRI (brain scans) to look at emoticons which at this point are perhaps the most simplified signs for faces we use.

Faces have been the focus of a lot of debate in cognitive neuroscience, particularly about the "face area" in the brain. One side says it's an area strictly devoted to processing human faces, the other side says that it's an "expertise" area and it activates because humans are experts at recognizing faces. It is one of the most fiery debates in cognitive neuroscience, and learned about in most all intro classes.

Amazingly, this study shows no activation of this "face area" when looking at emoticons. :-O

Using fMRI, the authors compare Asian style emoticons (non rotated) with averaged faces (photos of multiple faces that have been blended to be more "generic") that were expressing the emotions of happiness and sadness. Emoticons appeared first on their own, and in a second study embedded within sentences, while non-emoticon signs using the same characters were also used as fillers (i.e. ":O*-<").

They found that photos of faces activate both areas pertaining to emotional valence (right inferior frontal gyrus) and facial recognition (right fusiform gyrus), while emoticons only activate emotional areas but not face areas. That is, as the authors say, "Remarkably, emoticons convey emotions without cognition of faces."

This finding has very interesting consequences for understanding how brains process varying degrees of complexity in images. The implication here at least is that more simplified faces become tied more explicitly to a "symbolic" meaning as opposed to their iconic meaning of resembling what they look like. That is, more simplified images strip down the meaning to its core meaning disconnected to the iconic reference that they are framed within.

It would be interesting to see a graded approach to this — such as taking different degrees of representation from McCloud's gradient of "cartoonification" (or to use my term, "haplosis"). Are there different degrees of activation for different representations? Does activation for the fusiform gyrus all suddenly drop off at a certain level of simplification? How does this affect the debate that the fusiform gyrus is an "expertise" area rather than a face area?

(^_^)

Cartoony vs. Realistic Images in the Brain

Choi, Yeojeong, Kim, Takhwan, & Jaeseung Jeong. 2008. "EEG Source Localization during Empathy of Iconic and Realistic Cartoon Characters." Organization for Human Brain Mapping (OHBM), Melbourne, Australia ,15-19 June, 2008

In McCloud's Understanding Comics he proposed his theory of "cartoon identification" that cartoony* images are "identified" with better than realistic images. This study (pdf) tested McCloud's theory by using behavioral measures of a 7-point rating and EEG measures of the brain's electrical activity.

I've found that this theory of McCloud's was a bit ambiguous, since people have interpreted it in two different ways. It can either mean that people "identify" with cartoony images meaning...

1) They are perceived cognitively at a more "base" level.
2) That they empathize with the characters more.

Critics have usually tapped into the second reading, since it is close to a claim about how people "identify" with characters in a literary sense. However, I've always been more partial to the first interpretation (which I attempted to codify further in my book), though the two views aren't necessarily mutually exclusive. Indeed, cartoony images could evoke more empathy because they are more conceptually basic. This study aimed to examine the second version, strictly with a view of the degree of "empathy" styles create.

The experimenters contrasted comic strips that featured two opposing characters who were depicted in either realistic or cartoony styles (as in the example above) and put into different scenarios to evoke reader sympathy via who wins the confrontation (strips had varying endings, ex. winner is congratulated by a woman vs. loser is consoled by a woman). Participants only viewed one depiction, and only one option for the ending.

They then compared ratings in a behavioral study on a 7-point scale measuring empathy to both "winners" and "losers", and in a separate population, measured EEG brainwaves for the same stimuli.

Brain areas were activated that related to social perception, recognizing facial expressions, and seeing another person's pain**. They found both higher behavioral ratings of empathy and greater activation in the brain for areas for the cartoony characters than the realistic characters for both "winners" and "losers" (though different brain areas for different roles). They take these results to be support for McCloud's theory of identification that indeed, cartoony images do invoke greater empathy from a reader than realistic images.

NOTES:

*The authors, and McCloud, often use "iconic" to mean "cartoony" — I'm going to avoid this because it doesn't accurately convey what "iconic" means in a semiotic sense (i.e. meaning through resemblance). Technically, both cartoony and realistic images are "iconic."

** Just a caveat for those who actually follow the link to the pdf poster. The study shows nice pretty pictures of brains with activated regions to support its hypothesis, but these can be misleading given the actual methods used. Unlike a technique like fMRI, EEG does not give much information about where in the brain something occurs, and is much better at when it appears (i.e. the timing of processing). These brain images and results were gained using a "source localization" procedure which extrapolated from the data what brain regions were being used, a technique that is commonly employed, but often controversial. This isn't to say that the results are inaccurate, but they should be understood with this context.

Brainwaves for non-sequitur visual sequences

West, W. Caroline, and Phil Holcomb. 2002. Event-related potentials during discourse-level semantic integration of complex pictures. Cognitive Brain Research 13:363-375.

This study examines the neurocognitive processes involved with comprehending a series of pictures, like in comics. The experimenters pulled frames from an animated movie to create static picture sequences. There were two possible endings for each sequence: one with a normal ending, and one with a non-sequitur panel that did not make sense.

Comparison of these sequences used a technique called "event-related potentials" (ERP) that examines people's brainwaves with an EEG recording. The electrical field is measured off the top of the scalp through an electrode cap (like in hospitals), and by averaging out the noise at the critical point (the "event" — here the last panel) it can give you a nice smooth waveform that can tell you about the nature of the cognitive process. Unlike fMRI, ERPs don't tell you much about "where" in the brain things happen, but they do tell you a lot about "when" and a little about the nature of the process.

In this case, your brain distinguishes the difference in processing at less than half a second. The result was a "negative" deflection of the waveform roughly 400 milliseconds after the final panel appeared on the screen (panels appeared one-by-one). These waveforms are from the frontal right part of the head:



The BLUE line represents the normal sequence ending, the RED line the non-sequitur ending. Note that the lines separate and there is a bump labeled "N400" that shows the processing difference (negative is up here). Because of the separation, we can tell that the brain is working harder to process the non-sequitur panel. If it was treated the same, the lines would stay together, like at the beginning of the waveforms.

This N400 also appears in language under similar conditions: where the brain is working harder to integrate semantic information into a meaning, though with language it appears in different locations on the scalp (more back of the head than front). In fact, the first paper that found an N400 for language used this same manipulation: comparing normal and incongruous words at the end of a sentence.

Unfortunately, more experiments of this sort have not really been done with sequential images. Fortunately, it's only a matter of months until I do more. Phil Holcomb, one of the authors, is also one of my advisors. My upcoming projects will be doing these types of brainwave studies using more targeted manipulations of the visual grammar.

Comics and the Brain... almost

Nagai, Masayoshi, Nobutaka Endo, and Kumada Takatsune. 2007. "Measuring Brain Activities Related to Understanding Using near-Infrared Spectroscopy (Nirs)." In Human Interface and the Management of Information. Methods, Techniques and Tools in Information Design, 884-93. Heidelberg: Springer Berlin

Looks like I was beat to the punch... I've found a study from last year that analyzes the activity in the brain while reading comics. However, it doesn't say much.

The authors use near-infrared spectroscopy to measure blood flow in the brain while reading comics. This technique uses infrared light to measure where blood flows in the brain, which can thus indicate the brain regions involved in various behaviors. They find that "the left prefrontal lobe region is activated when people actively try to understand the comic stories and to memorize their contents for reporting in the future."

However, there are extensive problems with this study. First, the number of stimuli they use is extremely small (only 6 strips) as is their population (13 people... which does not add up to counterbalancing). Comparatively, the study I'm planning will use 180 stimuli per trial (720 strips total) and use somewhere from 24 to 36 people.

Additionally, the increase in blood flow that they observe only occurs in "reported" conditions — where subjects are actively making a judgement about the stimuli, as opposed to scenarios when they are not. This seems more to reflect the well-reported cognitive activation for making judgements than anything about the structure of the comics themselves.

So... this really doesn't tell us much about comics and the brain, but its nice to see other people are at least taking stabs at it as well.

America vs. Japan: Brains and Comic/Manga Panels

Via the TCJ message board, Nathan has pointed to an article in the Boston Globe that discusses the differences in brain activation between "Eastern and Western" perceptual processing. The study claims that "Westerners tend to focus on central objects more than on their surroundings" while Easterners "tend to focus more on the context as well as the object." From the article:

To use a camera analogy, "the Americans are more zoom and the East Asians are more panoramic," said Dr. Denise Park of the Center for Brain Health at the University of Texas in Dallas. "The Easterner probably sees more, and the Westerner probably sees less, but in more detail."

"Literally, our data suggest that people see different elements of pictures," Park said. "If you're looking at an elephant in the jungle, the Westerner will focus on the elephant and the Easterner is going to be more thinking about the jungle scene that has the elephant in it."

In a way, these findings are supportive of McCloud's claims that manga use more "wandering eye" type of panel "transitions." The evidence from my own more formal study comparing panels from Japanese and American comics (in my paper Cross-Cultural Space) seems to support this conclusion... somewhat.

My study found that American comics by far used more comic panels that featured a whole scene ("Macros"), while Japanese manga used equal amounts of panels with whole scenes and individual characters ("Monos"). Manga also used a great deal more "Micro" panels, which feature a "zoom."**

These results would seem to support a view that Japanese panels allow a focus on the broader environment, since they are breaking up the single environment into smaller parts. However those smaller parts are giving focus to the smaller parts instead of to the larger whole. So, in a way, manga panels are getting both the environment and the detail of the objects.

Unfortunately, my coding in this study was a little deficient, since at the time I lacked an "Amorphic" category that contains purely environmental information. These panels were coded as Micros at the time, but really should be their own category. On the plus side, I now have a larger and more diverse sample of comics to code and a richer coding scheme, I just need to get the peoplepower to do it (read: undergrad research assistants).

Update: An additional thought I just had related to this is the extant to which these claims are generalizable into two categories of East vs. West. At least regarding the graphic form, will we find that American books are the same/different as various European books? Can Japanese manga really be lumped in with Chinese, Korean, and other Asian comics' structure? Perhaps we'll find that there's a lot more diversity out there than we suspect...

**(The graph above shows a reanalysis of these numbers, getting rid of two American books that had "high manga influence" — the difference is slight but significant. Check the paper for initial interpretation/numbers)

Event segmentation in panels

I've been reading up lately on research related to how people segment events and their boundaries, particularly the brain areas associated with their processing through fMRI. In one study, they first showed people videos of events, then on subsequent trials asked them to identify fine-grained and coarse-grained event boundaries. In all trials, they found brain activation coinciding with the boundaries that were identified.

The results support a hypothesis that events are hierarchically organized, as fine and coarse grained responses in the passive viewing did yield differences. The brain activity in response to coarse grain event boundaries was stronger than for fine-grained boundaries, indicating modulation for hierarchical structure.

Reading this got me curious as to whether there are different cognitive effects for the representations within comics' panels for showing an event at different stages of its enaction. The "Marvel" style always pushed for people to be at an exaggerated state of the action, reflecting the event boundaries rather than their internal parts (I remember a vivid image from How to Draw Comics the Marvel Way with a drawing at all stages of a guy's punch).

Does the stage of the drawn action have an effect on cognitive reading? Since the Marvel exaggeration pushes the event to its boundary, does this mean that it may be processed easier, because it demarcates the segmentation as opposed to the fine-grained inner parts of the action? Or, would a drawn slice of an action create a boundary effect no matter what, since the more fine-grained parts are left out of the representation anyhow?

This would not be that hard an experiment to perform in fMRI, especially using comics as stimuli. The trick (as usual with experiments) would be creating sufficient stimuli that represent actions at different stages in their enaction. So, a punch would be shown in one condition at an exaggerated pose, and in another condition in a relatively unexaggerated pose. Would we find the same differences in fine vs. coarse grained processing of perception of event structure?

Coercion... of meaning!

Today I gave my big first year project presentation to the psychology department. From what everyone has said, it went very well. Of course, the project itself is still underway, and I will be running several more subjects in the lab, while probably continuing the study as a whole throught the summer. This shot is of me and my advisors from afterwards. (R to L: Ray Jackendoff, Phil Holcomb, Me, Gina Kuperberg):

As I've mentioned before, my talk involved looking at the "Event Related Potentials" or ERPs involved with processing a certain type of linguistic phenomenon called "semantic coercion." ERPs are a measure of the electrical activity of the brain. We don't get a good fix on specific brain areas that are at work, like in fMRI, but we do get very detailed analysis of the time course of events and certain waveforms do seem to indicate types of brain functioning in contrast to each other. We measure this electrical activity by sticking a cap of electrodes on people and feeding the signals into a computer, which then averages out the noise over several subjects and trials to give a smooth wave for time locked events. Here's me in the cap...

So, I looked at these brain waves for semantic coercion, which involves the extraction of "hidden" meaning from sentences like The chef finished the chicken before the main course. Someone can't literally "finish a chicken," they have to finish doing some action with it, like cooking. Since the event isn't stated outright, it's said to be "coerced" from the combination of the verb "finished" and the direct object "the chicken." Here's a waveform from one of the sites on that cap that I got in the experiment:

While this is interesting as a linguistic phenomenon, I think it's really just a warm up for more comic related studies. Since I couldn't resist, I even opened my talk by showing this strip:

Now, if you look carefully, coercion happens here too. We never see the event of Snoopy catching the ball, yet we know the event happens based on the information provided by the other panels. In addition to other things, coercion is perhaps one of the things that McCloud was trying to get at with his notion of "closure." In many ways, coercion here is an invisible meaning that is created out of the visible components of the graphic sequence. Graphically, it's the stuff that happens "out of view" of the panels. The problem is that McCloud extended this to the (linear) relationships between all image sequences, which just doesn't work.

So, if I do find anything fairly robust in the ERPs for verbal coercion, perhaps a study of visual language coercion could be on the horizon as well? Or perhaps a theoretical paper first...

The Brain

Not much is known about the Brain. Here's a nice little video explaining about it:

Essays on "Narrative Art"

Rob Vollmar has an ongoing essay up about "narrative art" on his blog, currently serialized in three parts:

Part One
Part Two
Part Three

It seems like he might be going somewhere with it, but as someone who has studied a little of cognitive neuroscience (and hopefully will be doing direct research on it very soon) I am a bit put off by his continued invocation of right/left brain distinctions. So little of the brain's functioning is known that it is easy to make broad sweeping claims about it and hard to say anything truly substantial. It might seem like a picky thing, but it struck a nerve for me...

It is easy to be enticed by the desire to discuss the brain. After all, it is the hidden key to understanding human activity, and I can see how mentioning it lends a feeling of legitimacy totalks of "narrative art." However, in most discussions (like here), it is largely irrelevant. "Word, images, and writing" can adequately be described and interestingly discussed as human behavior without invoking vague pop-psychological discussions of the brain, especially for his "historical" aims.

It is very hard to make claims about neurological activity (like that "narrative art" involves right or left brain activity and/or their interactions) without some sort of experimentation. Hell, it's hard to make conclusive claims about the brain even with experimentation! (…which is partially what makes it so intriguing to study)

At this point in studies about "narrative art," (as Vollmar calls it) just discussing the functions, of how image and text work together is enough to provide fascinating reading. Vollmar clearly has intuitions that can lend to interesting observations about this topic. I hope that his future writings can tap more directly into them.

In both scholarly and public avenues, I often get asked about "comics" and the brain. The fact of the matter is, no one knows anything (yet). I know of no studies addressing it at all (yet). At this point it is wholly conjecture, and a big blank white page on which to paint any number of discoveries (or tabulate data, as the case may be).

With that I will now turn to sleep, so I can wake tomorrow, begin this adventure called "grad school," and aim to hopefullly build some contributions to this neuro-comics discussion before too long.

Too Many Twos

I've got a short new piece up at Comixpedia for my ongoing "Comic Theory 101" column called "Two Many Twos." This one continues my series of short pieces where I try to illuminate issues involved with the intersection of “comic theory” and linguistics, here probing into the quandary called the "Problem of 2."

Like the last one, I've just tried to pose the issue and a little demonstrative visual puzzle, hoping that people might ponder/discuss it. Really, the whole thing just grew out of the last four panels, and I thought they'd be fun to work into a full piece.

I suppose this is also the first work I've done with my "avatar" as a narrator. Rather than turn myself into a kind of character, I've instead opted to keep it casual. I'd hoped for this to downplay the "me" in favor of the emphasizing the ideas more, but unfortunately in this case the ideas are demonstrated through the "me." Doh!

Buddhism and Brains

This month's Wired Magazine has an interesting article about Buddhist meditation and neuroscience. While I haven't read the actual paper, I can't say I'm particularly surprised that the study showed a huge increase in gamma waves and altered brain structure. This would have been perfect for some of my classes as an undergrad, where much of my focus was on "Buddhist Psychology."

A lot of the critiques that are discussed in the article largely stem from a misunderstanding of Buddhism (aside from the worrying about getting acurate and unbiased results). While in some sects and practices, Buddhism is very much on par with the orientation of other religions (such as Pure Land Buddhism), but in other ways though, it is far closer to science and psychology. Indeed, some aspects of Buddhism actually reveal the limitations and problems rooted in scientific dogma, like the belief in objectivism (although, I'd argue that it comes from a root in religion anyhow).

My undergrad thesis advisor co-authored an interesting book on the relations between Buddhism and cognitive science that discusses a lot of these same issues. I imagine we'll be seeing more books of this sort over the next few years.