Comics and the Brain... almost

Nagai, Masayoshi, Nobutaka Endo, and Kumada Takatsune. 2007. "Measuring Brain Activities Related to Understanding Using near-Infrared Spectroscopy (Nirs)." In Human Interface and the Management of Information. Methods, Techniques and Tools in Information Design, 884-93. Heidelberg: Springer Berlin

Looks like I was beat to the punch... I've found a study from last year that analyzes the activity in the brain while reading comics. However, it doesn't say much.

The authors use near-infrared spectroscopy to measure blood flow in the brain while reading comics. This technique uses infrared light to measure where blood flows in the brain, which can thus indicate the brain regions involved in various behaviors. They find that "the left prefrontal lobe region is activated when people actively try to understand the comic stories and to memorize their contents for reporting in the future."

However, there are extensive problems with this study. First, the number of stimuli they use is extremely small (only 6 strips) as is their population (13 people... which does not add up to counterbalancing). Comparatively, the study I'm planning will use 180 stimuli per trial (720 strips total) and use somewhere from 24 to 36 people.

Additionally, the increase in blood flow that they observe only occurs in "reported" conditions — where subjects are actively making a judgement about the stimuli, as opposed to scenarios when they are not. This seems more to reflect the well-reported cognitive activation for making judgements than anything about the structure of the comics themselves.

So... this really doesn't tell us much about comics and the brain, but its nice to see other people are at least taking stabs at it as well.

Comics as a Binary Language

Laraudogoitia, Jon Pérez. 2008. The Comic as a Binary Language: An Hypothesis on Comic Structure. Journal of Quantitative Linguistics 15 (2):111-135.

This study examines the structure of comics by converting the contents of panels into binary code. Coding a broad number of Eurpoean comics, a panel holding the protagonist of a story ("lead character") is given a "+" while a panel without is given a "-". The author then uses a series of computations to examine the regularity of sequences where the protagonist does or does not appear, or if there is constancy to the amount that they appear througout a book.

The results show that there is a quasi-regularity to sequences that feature the protagonist or don't feature the protagonist. That is, there are "runs" of sequences with protagonists, then runs without.

While interesting for coming up with a positive result — and very creative for applying computational methods to comics (somehting I don't think has otherwise been done), I find numerous problems with this paper.

First, why should we assume that Protagonist vs. Non-protagonist is a meaningful binary juxtaposition? In some ways it reflects of my distinction between Active and Inactive (or Passive) entities in a panel (originally based on Natsume's distinction of "positive" vs. "negative" entities). However, my breakdown is superficially "things that move across panels" to "things that don't." Protagonists could fall into either one of those categories given the appropriate sequence.

But... what if there is more than one protagonist? What if a scene shift happens where a new character becomes the lead character — this would just be coded as a consistent "-"?

Mostly though, I am unsure of what is interesting about these results. The visual language in comics features consistent "runs" of protagonist or non-protagonist panels: so what?

The analysis throughout focuses only on linear sequences based largely on Markovian chains, but I think my work has strived to show that sequences of images cannot simply be considered linear sequences. They have hierarchic structures guiding them — which such a binary analysis of the surface elements would be unable to show.

This study is an interesting first attempt at using computational methods to analyze visual language structure — and I love that the research has now begun permeating such extants. Hopefully further studies will bring more interesting results.

Coherence-building in comics

Bridgeman, Teresa. 2004. Keeping an eye on things: attention, tracking, and coherence-building. Belphégor 4 (1).

Bridgeman's article discusses various aspects of coherence-building throughout comics structure — "coherence" being the discourse theory notion of a salience across various units. It thus joins various other works that apply discourse theory to comics, though dabbles in cognitive science a bit as well.

The piece covers a lot of ground over various parts of comic structure (style, color, composition, layout, etc), and it uses cognitive principles to at least elucidate the elements of structure fairly well, though it sticks to a fairly generalized notion of them. On the whole though, not much is "new" about the work presented here. It takes the fairly overt elements of structure and simply maps cognitive-theory-lite to them, while also drawing from a well-done mixture of McCloud and Groensteen's ideas.

McCloud and Groensteen's theoretical orientations are often put at odds with each other, yet this paper makes ample use of both of their theories. While I may not particularly subscribe to either of their theories (I do have my own ideas, you know), it's at least nice to see that not everybody falls into one camp or the other.

Partly though, the non-novel nature of the paper may be due to "intent," which is less to provide a cognitive analysis of the structure, so much as (it seems) to use cognitive principles for analysis. This seems to be an inherent disciplinary tension though. While I do think it succeeds as an application of cognitive theory to literary analysis (which most of the paper is devoted to), I'm also wary for whether it knows the difference between the two intentions.

Connectedness in comics

Weber, Heinz J. 1989. Elements of Text-Based and Image-Based Connectedness in Comic Stories, and Some Analogies to Cinema and Written Text. Paper read at Text and Connectedness: Proceedings of the Conference on Connexity and Coherence, July 16-21 1984, at Urbino, Italy.

Weber attempts to create a textual model of comic communication drawing on cinema studies and text research (similar to the intents of Saraceni and others).

He describes three sections: Graphic, cinematographic, and textual, as well as the intersections between them. He also postulates several degrees of “connectedness” ranging from conformity, sequential and integrated connexity, cohesion, and coherence.

Conformity deals with arrangement of panels – conventionalized formats, while connexity can either relate to the internal relationships of elements within panels, layout issues, divisional panels, metonymic panels, the shifting of a balloon's tail to different roots, etc. Cohesion depends on causality between the succession of panels (syntax/semantics). Finally, coherence deals with pragmatic relations between “text external” elements.

Like other papers of this ilk, it provides a broad scale analysis of aspects of comics' structure, yet doesn't delineate them carefully enough to detail the componential role they might play. Instead, it uses overarching "principles" to tie them all together to create a goolash of comic structure.

Eye-movements reading comic pages

Omori, Takahide, Taku Ishii, and Keiko Kurata. 2004. Eye catchers in comics: Controlling eye movements in reading pictorial and textual media. In 28th International Congress of Psychology. Beijing, China.

A team of Japanese researchers perform two experiments examining eye-movements across comic pages to show that both page layout and balloon placement factor into how readily comic pages are read.

They found that, for an average of 8.5 panels per page, there are an average of 20.3 fixations. Most of their study focuses on panels that were skipped over for one reason or another, and examining modifications made to see whether they would still be skipped over.

There were two major changes that showed significant effects in decreasing the rates that they were skipped: balloon position and panel layout.

The first factor in skipping is if a panel is followed by another panel with dense text. They altered the "dense balloon panels" by distancing the balloon further away from the preceding panel. This change resulted in a significant reduction in the times that the preceding panel was skipped.

The other major factor was when panels were vertically stacked next to a long adjacent panel (what I call "blockage"). The lower panel was often skipped so the reading follows the horizontal path. When altering these layouts to make the panels horizontally arranged, the rate of skipping decreased. However, this phenomenon was only observed in a couple of scenarios (6 instances) and they don't mention how many of these skips lead to going back and rereading the skipped panel. They also don't state how many times "blockage" occured and didn't lead to skips.

Slight decreases in skipping were shown for moving characters' positions within a panel, though not to high percentages (significance is not shown).

Additionally, a recognition task asking whether various panels were or were not in the comic showed significant increases in accuracy for the modified versions. No differences were shown in accuracy of reading comprehension for the story.

While they state that their participants all had comic reading experience, I wonder the degree of "comic fluency" that they have. The desire to jump towards panels with dense text insinuates a focus more on text than on the visuals, which was characteristic of a naive comic reader's eye-movements compared with an expert reader in Nakazawa's eye-tracking study.

Further, this study supports an idea that "blockage" situations are harder to process (evidenced by the skipped panels). However, I have empirical evidence from my own experiments on page layout (to be posted soon hopefully) that following the vertical path of panels is the prefered reading path, and that preference for it does depend at least partially on expertise in comic reading. Also, their studies used only the results from 25 subjects (half seeing modified versions half not), whereas mine used 145, so looking at a broader populace would be good here.

Hypothetically, I could tackle this issue myself, since the lab next door to mine has an eye-tracker at my disposal. We'll see... I have a few other things on my plate right now.

Children Interpret a Comic

Pallenik, M. J. (1986). A Gunman in Town! Children Interpret a Comic Book. Studies in the Anthropology of Visual Communication, 3(1), 38-51.

This insightful article examines children’s understandings of comic books over time using a Western comic A Gunman in Town!. The study looked at ten children in each of 3rd, 5th, and 8th grade, balanced for gender and race with diverse socio-economic status. They were shown each frame individually and asked its contents following each panel. This might not have hugely hampered the sequential understanding though, since the panels seem largely dominated by text.

All the children recognized broader information: that the book was a Western and that it would end with the villain losing. All the children were concerned with the concept that the story was going to end, showing knowledge of it as a story and that stories have endings. Most of the phenomena showed small jumps and differences in understanding between grades. For instance, in readings of the last panel of the book, a steady increase of children recognized the correct reading order of word balloons (Grade:number of kids – 3rd:2, 5th:4, 8th:7).

Many third graders would skip over reading dialogue, especially when it was heavy in panels. They also will gather most of their reading from stereotypic knowledge, missing important story elements or filling in missed information with further stereotypic knowledge about genre.

Fifth graders pick up far more information than third graders, with explanations seeming less stereotypic – allowing them to anticipate and integrate events more quickly and accurately. Eighth graders “move back and forth between their knowledge of conventional genre structure and the particular story” (46). Fifth graders are more capable of predicting future events from individual panels — each panel implies something about future events. While eighth graders can predict to the end of the story, fifth graders make more short-term predictions about action sequences.

Eighth graders see the story as conventionally ordered by the dictates of the genre. Two strategies were used by eighth graders. When uninterested, they use a “flat” style that perceives and decodes the story as it unfolds bit by bit. A contiguous reading style incorporates the understanding of the genre to expand on the given information with schematic knowledge (unlike with third graders, this isn’t to make up for missed information though).

These results further indicate that the ability to understand sequential images increases with age, and perhaps with exposure/experience.

The Comic Strip and Film Language

Lacassin, Francis. 1972. The Comic Strip and Film Language. Film Quarterly, 26(1), 11-23.

In this piece translated by David Kunzle, French theorist Francis Lacassin discusses the similarities between the "syntax" of film and comics, noting that they both use "shots" as their base units. For him, this includes various things, like various degrees of framing (long shot, close up, medium shot), dynamic use of what could be multipanel representations ("panning"), as well as semantic alterations, like subjective viewpoints. (I would argue that this isn't "syntax" at all... but that's a larger post).

He argues that though film and comics emerged around the same time, these techniques came first in comics — not the other way around, as is often argued — and that they may have been autonomous developments not influencing each other at all.

He writes: "It is more reasonable to suppose that comic strip and cinema have both separately drawn the elements of their respective languages from the common stock accumulated in the course of the centuries by the plastic and graphic arts." (14)

To this I would question, is it really through historical development, or is this just a reflection of the structuring of people's minds/brains?

He hypothesizes also that film and comics both accomplish their sequential meaning by use of the film theory of montage, which for Lacassin appears to cover most things that do not appear similar to real-world perception.

In his own section at the end, Kunzle criticizes Lacassin for claiming comics were invented before cinema, while those framing techniques are cited being used by authors two generations before that "birth" of comics. Kunzle then discusses the work of 1800s artists Töpffer, Doré, and Busch, noting that they used various techniques like close-ups and polymorphic representation (where one character is repeated in a single frame showing the unfolding of action), among others.

Thought Bubbles, Children, and Autism

These are several articles I found particularly interesting looking at the understanding of thought bubbles, and using them to help autistic children.

Wellman, H. M., Hollander, M., & Schult, C. A. (1996). Young Children’s Understanding of Thought Bubbles and Thoughts. Child Development, 67, 768-788.

Several experiments on 3 and 4 year olds show that thought bubbles are understood at fast rates by both age groups as depicting thoughts. This is interesting, because at similar ages, other visual conventions such as speed lines are acquired over age (I'll post on these articles next maybe).

This contrasts with the findings in the Yannicopoulou study, which showed that preliterate children could not distinguish speech from thought balloons – which was not tested in these experiments. They suggest that this shows evidence for “something recognizable from our everyday understanding or experience of thoughts themselves” tapping into "theory of mind" knowledge...

Wellman, H. M., Baron-Cohen, S., Caswell, R., Gomez, J. C., Swettenham, J., Toye, E., et al. (2002). Thought-bubbles help children with autism acquire an alternative to a theory of mind. Autism, 6(4), 343-363.

Children with autism have specific difficulties understanding complex mental states in other people like thought, belief, and false belief and their effects on behavior (what are known in psychology as "Theory of mind").

These children benefit from focused teaching about thoughts, where beliefs are likened to photographs-in-the-head. Here two studies, one with seven participants and one with 10, tested a picture-in-the-head strategy for dealing with thoughts and behaviour by teaching children with autism about cartoon thought bubbles as a device for representing such mental states.

This device led children with autism to pass not only false belief tests, but also related theory of mind tests. These results confirm earlier findings of the efficacy of picture-in-the-head teaching about mental states, but go further in showing that thought-bubble training more easily extends to children’s understanding of thoughts (not just behaviour) and to enhanced performance on several transfer tasks. Thought-bubbles provide a theoretically interesting as well as especially easy and effective teaching technique.

Pre-literate understanding of speech balloons

Yannicopoulou, A. (2004). Visual Aspects of Written Texts: Preschoolers View Comics. L1- Educational Studies in Language and Literature, 4, 169-181.

This study assessed preschool aged children’s ability to understand various features of Carriers (i.e. thought bubbles and speech balloons), despite not being able to understand written language. Tests showed:

- 87.1% recognized angular balloons meant anger.
- 83.7% correlated a flowery border to politeness.
- 78.7% recognized increase in volume by increase in size.
- Speech vs. thought balloons were distinguished for their meaning at chance (49.7% speech/47.5% thought)
- Most poor was recognition of other languages as indicated by variation in text.

These results were fairly interesting, especially since they imply that children can recognize aspects of manner of speaking (politeness, anger, etc), yet can't differentiate plain thoughts versus speech. Part of this might relate to a general developmental trajectory though — that children don't yet have "theory of mind," the recognition that other individuals' have thoughts of their own. Preschool children are roughly at the age where this ability is developing, so the problems they had recognizing thought bubbles might be due to their lack of understanding thoughts in other people in general.

However, these data contrast with other studies in this regard that I'll be posting sometime soon.

Review: The Language of Comics by Mario Saraceni

Saraceni, Mario. 2003. The Language of Comics. New York, NY: Routeledge.

Saraceni's The Language of Comics is one of the few books that attempts to present a holistic theory of how comics work, and draws upon work from "applied linguistics" no less. The book is actually a stripped down version of his dissertation (as is his article "Relatedness" from the Graphic Novel collection). (And good luck finding the dissertation... I had to print it from microfilm on interlibrary loan).

Unfortunately, much of his approach seems to feel of grafting McCloud's work to ideas in applied linguistics in a simplistic (and uncited) way. For instance, he proposes a gradation between semiotic types (like symbols and icons), and can well be compared to McCloud’s Big Triangle.

He treats the sequential aspect of panels as equal to sentences, giving them a “discourse theory” type analysis (like the dissertation by Stainbrook). Saraceni claims meaning is created through commonality between elements in panels, alternating with successive new and given information. He also uses "semantic fields" (connected meanings: like how "snow, caroling, pine trees, and presents" invokes "Christmas") to unite panels not encompassed by this information structure.

However, in doing so, he eschews the role of linear sequentiality, yet provides no argument for why people do indeed read in consistent sequences. The result is essentially a watered down version of McCloud's closure — which it is: his dissertation has the theory in full, and exactly does shoehorn discourse theory onto McCloud's transitions. Some of his insights here are useful and enlightening, yet they deal entirely with "exceptions." He rarely discusses "run of the mill" things like the depictions of events, instead culling his examples from very experimental comics storytelling, like Peter Kuper's The System.

Other chapters cover things like word balloons (perceived as equivalents to direct quotes) and drawings of eyes to buttress a discussion of subjective and objective viewpoints. The final chapter is about computers, which seems out of the blue and has next to nothing to do with comics.

Since it uses applied linguistics, much of the book feels attuned to what might be useful for literary studies. To this extant it might work very well. However, as a theory of "meaning" it falls short, largely because it does not address any type of cognitive system, and lacks even McCloud's precision of surface categorization.

My biggest gripe about the book is that it is presented as an introductory textbook as part of the Intertext textbook series, has no citations outside a "recommended reading" list in the end, and is written with a matter of fact tone that presents it as an authoritative stance on the body of knowledge of this field. The truth is that no such body of knowledge exists at this point for "comics theory." Right now, we're in that period of science where lots of different viewpoints are popping up, just waiting for an encompassing paradigm shift to sweep in and take over.

Even if I were to come out with a book of my full theories, it wouldn't be a de facto textbook because it would be my views drawing upon that body of research. As a result, to those "in the know," the format and style make this book seem misleading in its intents for fronting Saraceni's views as well established scholarship.

To end on a good note: Though I think it fails in not using a cognitive approach, I do like that the book tries to use concepts from linguistics with "comics." It shows that this type of approach is not just intuitive to me, but to others as well, and locating the book in the broader field of linguisics is good for the field as a whole.

More on Imitation in Drawing

Brent Wilson and Marjorie Wilson have done a lot of great work on child drawing that has influenced my thinking, especially since they take comics and manga into account a lot. I got to meet Brent last year at a manga exhibit, and he was a really nice guy. Here's another good (old) article that I recently read (it won't be the last):

Wilson, Brent, and Marjorie Wilson. "An Iconoclastic View of the Imagery Sources in the Drawings of Young People." Art Education 30.1 (1977): 4-12.

Wilson and Wilson present findings that contrast the century-old belief that imitation is bad for learning to draw. This view focused on the belief that children have an innate purity to drawing that emerges out of their natural tendencies. Imitation is thought to defile this purity. Additionally, due to their iconicity, drawing has been seen as the correspondence between the world and mental equivalences of those objects.

The Wilsons' data counteracts this with the observations that of the hundreds of drawings gathered from high school students, virtually all of them could be traced back to imitation of some other source of representation (especialy comics and cartoons, but not much fine-art). They note that the learning of drawing might be more similar to learning words (though they don't seem to really know what that means beyond common sense knowledge).

They propose that people are using/creating mental models for drawing, and that minor modifications to generalized structures can aid in creating specific representations. At times, these models would be begun to be employed, yet abandoned part way through if the drawer couldn't produce the desired result with that schema.

People might also have mental models for one type of representation, but be unable to do drawings outside of that model. So, drawing novel objects either results in sub-adaquate ability or deploy existing models from other domains.

In total, people seem to be able to store hundreds of these types of mental models. For instance, one subject who drew comics could create figures in innumerable ways. When drawing a figure he hasn't before, they hypothesize that he "averages" several of the models together to produce the new form. To me, this raises the question whether the aggregation of these models creates a singular more abstract schema or whether it remains a catalogue of numerous "malleable" models.

---
This means the mind has a place to store such visual schemas, a "photological" component akin to a "phonological" component for spoken language. As I've said before, if perception is the desired stimulus for drawing, models aren't created from existing models — so the system never creates a conventional set of signs.

And just to riff on my previous post, this is also the reason why drawing perspective might be "awkward" for learners — because it sidesteps metal modeling in order to exact a system of depth through measurement (as opposed to schemas... though "loose" drawing of perspective might involve some level of mental modeling created by learning how to do it. I'll have to see if there's research on that...).

I had a 13-ish year old student when I taught "drawing comics" in an afterschool program, who could not draw perspective for a tile floor in a hallway. He drew the receding lines of the hall walls with us as a group, then when drawing the floor on his own he drew it flat — like an aerial view. My interpretation: he couldn't override his existing mental models for spatial representation with new ones for perspective. That's not to say he couldn't if he worked at it, but at that moment he couldn't do it.

Finally, note that perspective and schemas are both learned — the difference is that one is acquired "effortlessly" through imitation (as they'd say in language acquisition) and the other is taught explicitly.

Point perspective

Austin Kleon has a nice little post about point perspective in comics, noting his like of artists who don't use it at all (link via Derik). The preference for point perspective is of course wrapped up in the whole desire for iconicity that readers of this blog are probably sick of hearing me rant about.

His post got me to think about some other related issues. For instance, point perspective was developed in the Renaissance, which I imagine coincided with the Enlightenment's focus on discovery about the world and the rise of science. (Though, I have no idea since I'm not an art historian.)

What is worth remembering though is that point perspective was truly a discovery. The human mind/brain may be able to see in perspective, but we don't draw in it. There was a study** I read that talked about South African children (ages 5-9) who had trouble with understanding certain parts of images. The parts they misinterpreted the most were perspective, depth, and shading — all highly iconic and things that must be explicitly taught to people learning to draw.

Interestingly, when looking at the data, the means for misinterpretation drop for children at Grade 3 (and 9 yr. olds) in almost all categories. The conclusion of the author is that schooling teaches children how to understand images, but this could just be a coincidence in that children’s exposure to images comes in a school setting. That is, it’s not about instruction, but about exposure.

Whatever the case, perspective is not a built in part of the human graphic system. This again goes to the point that drawing is less about mimicking the perception of the world as piped through an individual's mind, and more about the way minds are enabled to convey concepts visually.

Update: I feel I should add, that there's nothing wrong with learning how to draw with point perspective, only that our minds' graphic system is not predisposed to it. As an academic, I'm not prescribing anything, just analyzing. Learning perspective requires iconic understanding that doesn't just come out of imitation of other people's drawings. That is, it once again skirts conventionality and the establishment of mental models for drawing in lieu of imitating perception.



** Liddell, Christine. 1997. Every Picture Tells a Story—Or does it?: Young South African Children Interpreting Pictures. Journal of Cross-Cultural Psychology, Vol. 28, No. 3. Pp. 266-283

Review: "Copying and Artistic Behaviors"

Smith, N. R. (1985). "Copying and Artistic Behaviors: Children and Comic Strips." Studies in Art Education 26(3): 147-156

Smith argues that the negative views on "copying" demonstrated by art educators since the 50s is misplaced in some contexts. She claims that some forms of copying are good, and the relative value of copying is based on three factors: need, model, and process. She examines varying fields through use of a corpus of comics produced by American children, noting that themes and genres are copied greatly. She didn't find that the children copied the drawing style as much.

My curiosity is whether this is due to lack of practice/exposure though. The examples given by a child with "unusual ability" seem hardly on par with Japanese drawings of children of the same age that copy manga en masse. This child did copy various elements of drawings, though not absolutely. For instance, when copying Charlie Brown, he imitated parts but altered/left out others. Another child drew the typical "lumpy" figure of Captain America to show his musculature. Smith conjectures that his intent was to draw someone "strong" as opposed to drawing a bicep in particular.

To this extant, these children's copying seems to be drawing characters/features to the point of recognition — not iconic match. In other words, they're trying to convey concepts visually, not create "realistic" pictures (or even "accurately" imitated images).

While interesting to see much support given to imitation, most of it is not structural, and still maintains an "Art" perspective. The "need" assigned to copying is largely social or emotional/psychological, not structural or cognitive. (For instance, it says imitation suits a child's need to "play out" conflict in fantasy, as opposed to saying that children copy because their brains are pattern seeking machines).

Social need is Language-like though, as it heralds conventionality. She also marks copying as important as a natural behavior in socialization, since "younger children initiate copying as a means of acquiring desired knowledge" while "older children want to master images representative of their culture" (147).

Also interesting was her statement why she wanted to look at comics in the first place: "Comic strips are of interest because children frequently and spontaneously initiate copying of them despite disapproval" (148). No citation is given to this statement, but are comics copied more than other forms of visual communcations in culture? (it wouldn't surprise me if the answer is "yes") And, if so, doesn't that say something about the structure of the stimuli in relation to the human mind — like maybe these signs are somehow attuned to acquisition and socialization?

Manga Literacy

I've made a couple additions to the Reference Bibliography, including this one:

Nakazawa, Jun. 2005. Development of Manga (Comic Book) Literacy in Children. In Shwalb, David, Jun Nakazawa, Barbara (Eds). Applied developmental psychology: Theory, practice, and research from Japan. Pp. 23-42

This English piece is a nice summary of the work of Japanese Psychologist Jun Nakazawa, as well as several other Japanese studies on "manga literacy." His various experiments cover a lot of ground, usually looking at students from 1st through 8th grade. Most all his findings show increased understandings with aging and expertise. I'll discuss only a few of the many studies in it.

The study I liked the most asked children to arrange randomly given four panels into a strip, finding that correct answers grew from fairly low for kindergarteners and 1st graders (5.2 and 6.6%) to high for 4th and 6th graders (around 80%). Another task on that test asked for students to fill in the blank of a missing panel, which no K/1st graders could get right with increasing percentages along older grades. Comparatively, adult college students were far better than the children.

He also has designed a "story comprehension" test to examine how fully they can recall plot aspects of a ten-page Doraemon manga. He showed again that the biggest growth came between 1st and 4th grades.

He also did some eye-tracking studies comparing the eye movements of an "expert" versus a "non-frequent" manga reader. The "non-expert" fixated far more on word balloons than images and had higher reading times. On the other hand, the "expert" reader made "fewer useless eye movements" that were smoother, in addition to a higher rate of skipping over more panels and balloons. However, the expert also had higher story comprehension recall than the non-expert, despite reading faster and skipping elements.

The second part of the paper looked a lot at the role of manga in education. One interesting finding showed that frequent reading of manga correlated to achievement in language arts (particularly sentence comprehension) and a liking of social sciences, though "not significantly with liking for art class." Several studies also indicated a higher comprehension for learning from manga than from pure textual "novelized" writing.

In all, the piece presents several very interesting findings related to children's (and some adults) understandings of manga, and it is a veritible treasure trove of citations and studies. It presents a "cognitive processing model" based on this work, though it's so general that it could apply to any type of media. Along those lines, it doesn't really break up understanding into any sort of "grammatical" components as I'd like to see, lumping in aspects of things together (like manga consisting of pictures, emblems, text, etc rather than breaking those things down). The best part of the paper is its overall picture: that the skills required to understand the "comic medium" are learned and increase over age and practice.

"Cartooning Symbolia"

Derik points to this comic, Cartooning Symbolia, by Dash Shaw that uses Mort Walkers terms from The Lexicon of Comicana for various symbolic elements of the graphic medium. It's an interesting experiment in formalism, and I especially appreciate the last twelve panels that introduce "new symbols," which well exemplifies the complete conventionality of symbols given that the reader has no idea what these things mean.

I read Walker's book for the first time last year, and here's what I wrote back then to the comixscholars list:

Its not a bad read. Its funny and lighthearted. One gag even got me to laugh out loud. He does point out an awful lot of graphic conventions used throughout many American comics – particularly strips – and makes a few interesting observations about them. Its by no means exhaustive, though it does have a surprising amount in it.

He also attaches a myriad of useless names to them, to the extant that you feel that his whole point for jargon is to be facetious (which it may well have been). You can tell that some of his terms have a logical origin to them, while others just seem made up because he wanted to give everything a name. I also have some discontent with his organization of these things, but for very specific structural reasons that I will bring up in some future writings of my own.

This organizational issue is related to my own attempts to compile a list of conventional signs in visual language. Many of the things he puts as separate signs I will include together in a larger category (like "smelly lines" and "sun rays" as both types of "path lines"). And while I'm at it... my list is ongoing, so please alert me to any more signs of this nature if you come across them. Perhaps someday I can compile them into more of a dictionary/wiki type project with examples and whatnot...

Timing

Newsrama hosts the first part of three articles about Time in sequential art written by Joanna Estep. The piece is very well presented, and I like how systematic her analysis is, especially her use of diagrams to push along the theory. It's well worth reading, and I look forward to seeing what her next installments bring.

However, I also want to point out that it makes certain assumptions that are largely passed on from the Eisner/McCloud tradition. Mainly, it holds that "one panel = one moment," which simply isn't the case if you actually look at sequences of images from books (as opposed to just mental theorizing – of which I've been guilty of too). There is nothing about two panels that dictates time is passing – only content that implies temporal succession can yield this result. And, once you see that many panel sequences don't inherently push time along, you realize that problems arise in any linear notions of time across panels.

Following this, it also reinforces the ideas that "spatial distance = temporal distance." I had some thoughts on this like four years ago that I've never really worked into a full-blown paper, but the basic idea is that panel sizes create a rhythmic structure for reading. To really see if this is true I'd need to do eye-tracking studies though…

I'll hopefully be posting an essay I've been working on about Time myself sometime soon, but till then my old essay Visual Syntactic Structures (and book Early Writings...) delves into these things for anyone interested.

Update: I now see that Timing Part 2 is posted too. Again, worth reading, but continues the assuptions in McCloud that "reading time = fictitous (i.e. mental) time." I'm also curious why she includes her "hierarchies within images" as being related to time, since she doesn't measure any increase or decrease thereof. I agree with this: I don't think foregrounding is related to time at all, though I do think its related to distinguishing things like who is the focused actor and who is subsidiary.

Update #2: Timing Part 3 is up now, rounding out the articles. This one is about the integration of text. I'm not sure what real relevance it has for the understanding of Time after stripping away the assumptions I talked about above, but she certainly has some interesting things to say about composition and reading orders. Go read.

Mayan Visual Language?

I haven't done a review for a while, so here's an absolutely fascinating one (again, listed in my bibliography):

Nielsen, Jesper, and Wichmann, Søren. 2000. America’s First Comics? Techniques, Contents, and Functions of Sequential Text-Image Pairings in the Classic Maya Period. In Comics and Culture: Analytical and Theoretical Approaches to Comics. Magnussen, Anne, and Hans-Christian Christiansen (Eds.).

This absolutely fascinating article provides a structural analysis of what could be interpreted as Mayan Visual Language. Some examples very clearly use the VL grammatical categories I've been researching, such as this one here (read here, R-to-L, click for high res.):



Most of these artifacts were taken from “vessels” (vases), so the sequentiality of the reading (layout) would be gained by turning the object itself. This is reminiscent of the 5000 year old goblet found in Tehran with sequential art on it. The authors also speculate on the usage of speed-lines and speech balloons, which have semantic variation in representation (speech balloons turn into flames used to show anger – a notable conceptual metaphor in its own right).

They also note writing and images exist sometimes exist independently of each other, but by and large are overshadowed by text-image pairings with sequential art. It's interesting the reverence placed on image-text pairings in contrast to Western counterparts:

"In Western society, the combination of text and image was, for centuries, considered a debased form of communication. Only artists who directed their work towards a mass audience, predominently the lower classes, dared venture into text-image pairings. The Mayas, however, considered the combination of text and image the most exquisite and exclusive form of artistic communication, and reserved it for elite consumption only." (p.73)

Would that we achieve what they had. All in all an absolutely amazing piece. I wish more analyses on cultural systems were done like this.

Kid's sequential drawings

This is a summary/review of an article I thought had particularly compelling evidence for why understanding sequential images is a learned trait. Highlights are all mine.

Wilson, Brent, and Marjorie Wilson. 1987. Pictorial Composition and Narrative Structure: Themes and the Creation of Meaning in the Drawings of Egyptian and Japanese Children. Visual Arts Research, Vol. 13, No. 2 (Issue 26). Illinois: University of Illinois

Narratives of urban Japanese children (manga) were compared to those of village Egyptian children. The argument was made that development differs based on graphically “rich” versus graphically “poor” environments. Egyptian children teach each other how to draw (“world of childhood graphic imagery rather than adult imagery” p. 10). Egyptian children’s drawings were floating, static, 2D, and symmetrical– more a reflection of perception (“intrinsic and intuitive bias towards simplicity”…”reflect humans’ innate preference for simple nonoverlapping shapes” p. 15). Japanese children’s drawings were often occluded, cropped, with lots of visual elements, had some sort of plane to ground the images, and employed “cinematic” techniques– similar to that found in manga (Japanese children by 8 or 9 may have passed the point where they are inspired by innate factors– p.15).

Over 2/3 Egyptian children drew narratives where the contents of one frame was not sequentially related to the next frame. All Japanese children drew sequential narratives – and at a “higher level of story structure” (p.16). “Japanese children were three or four times more likely to depict a related series of events or process.”

Their conclusion is that the urban versus village lifestyle, plus other cultural factors encouraging drawing are what lead to the difference in representational ability. My response would be that its not the urban/village lifestyles that cause this, but exposure to VL and practice with it. Japanese kids live in rich visual language culture (manga), and actively develop those this graphic fluency. They do note though, that Egyptian children did not have access to comics, and “television for the Egyptian children seems not to provide a functional model for producing the structure of graphic narrative plots” (p16). Manga, of course, does provide that for Japanese children.

This is another example of how looking at graphic creation through a Language perspective alters the way data is interpreted. Because drawings look like what they represent, the Art POV will attribute influence to all sorts of perceptual and societal influences. A Language perspective focuses mainly on the exposure and devlopment of those particular structures in their cultural surroundings: if you're going to produce (visual) language, what (visual) language is around you?

Gestures in comics

A doubleshot of reviews:

Fein, Ofer, and Kasher, Asa. 1996. How to do Things with Words and Gestures in Comics. Journal of Pragmatics, Vol. 26.6. Dec. 1996. Pp. 793-808

This study looked at the role of gestures in comics (specifically, those in the European comic Asterix). The study had people interpret the meanings of both panels from the comics, and of photos where people took on similar poses. The backgrounds of the panels were erased, so there was no context for the gestures. In one part, they were asked to write possible dialogue for the gestures, and in another task they were given potential meanings and asked to assign them.

It concludes that gestures in comics are interpreted the same as ‘real life’ gestures, and that the meaning imbued in them comes from the ingesticular force (i.e. the intent of the expression) rather than the propositional content of the accompanying speech (in word balloons). One interesting tidbit noted that some people said the photos were actually harder to interpret than the comics panels (though the stats disputed this). If this were true, then it would support McCloud's insinuation that cartoony images are more "base" than realistic ones. I'd like to know the VL fluency of the subjects and whether people with more "comics" experience rated higher or lower in this regard.

Raecke Jochen. 1999. Using Comics as Data for Research into the Connection between Pointing Gestures and Deictics. In E. André, M. Poesio, and H. Rieser (eds). Proceedings of the Workshop on Deixis, Demonstration, and Deictic Belief at ESSLLI XI.

Uses comics to analyze the relationship between deictics and gestures in Serbo-Croation. His method codes a corpus of comics comparing the relations of the images' gestures to the conent of the speech balloons. He finds that pointing gestures by far dominate the gestures, and pointing gestures alone do not fulfill the meaning of the representations (i.e. multimodality is necessary). This isn't surprising, since pointing gestures are indexical, which means that they only indicate meaing in something else (the same way a pronoun refers to a different element for meaning).

Subjectivity and a rant on Comics Scholarship

So, this is a review of a paper that is listed in my bibliography. If you'd like to see more of these, let me know...

Driest, Joris. 2005. Subjective Narration in Comics. Masters Thesis. Utrecht University.

This piece covers a broad overview of the ways in which “subjectivity” is represented in the “comics medium.” Of particular note is its analysis of word/thought balloons. The piece is largely influenced by film theory, and its relationship to comics and writing, however, it does not include citations to John Barber or other relevant works along those lines.

However, the piece didn’t seem to have a directed and focused hypothesis of sorts that it was out to prove. It hovered at the level of “these things are there” without probing that topic deeper. This isn’t necessarily terrible, considering that no other studies really cover this topic previously, except maybe in Saraceni 2000 & 2003 (which also weren’t cited).

It also seemed to comment upon a number of phenomena that occur in the “comics medium,” but didn’t seem relevant to the thesis (such as conventional graphic symbols). This is a trend I’ve noticed a lot in papers about VL & comics. Since they don't have an established a cannon of scholarship (or an recognized field to study them), people often feel the need to insert every interesting thought they have about it regardless of how pertinent it might be to what they’re actually writing about.

Relatedly, most essays on anything comics-related feel the need to define what a comic is in the paper – whether or not the paper is about how “comics” are defined. To me, this just seems to cry out an underlying complex that “nobody knows what comics are, so I need to define it.”

Guess what: whether they actively read comics or not, most everyone in our culture knows what a “comic” or “graphic novel” is to the extant that most scholars write about. People don't need to define what a novel or a film is every time they write about them, nor should they need to be told what a comic is. (Similarly, linguistics papers don’t define “language” in every paper – they just get to the meat of the issue).

At least from my perspective, the less exceptional we treat visual language and comics, the more they can be considered as equal with other forms of communication/literature. To invoke the metaphor: "Separate but equal" does not work, because it’s NOT equal. You need to have complete non-discrimination. By continually defining it where its not needed, “comics” (the social objects, and thereby the visual language associated with it) is implicitly placed into a “minority” position in the realm of criticism and scholarship.