Review: The Power of Comics

Duncan, Randy & Matthew J. Smith. 2009. The Power of Comics. New York: Continuum Books

The Power of Comics is a recently released “first textbook ever” for “comics studies”, authored by Randy Duncan and Matthew J. Smith (book website here).

Perhaps to be expected from a book on the general “comic studies”, it includes a broad range of topics, from the history of comic books to comprehension of the medium, to creators and fandom. In many ways, the vastly disparate topics brings into question the overall utility of conceiving of a singular field called “comic studies.” Indeed, a person interested in one particular subtopic may find the entire rest of the book irrelevant. No doubt though, the scope of the book is meant to be inclusive and to at least cast the possibility that such a broad field could exist.

As a textbook, it succeeds in content, scope, and execution. The chapters are well laid out, have thoughtful questions at their ends, and several chapters end with very practical examples of analysis that serve as models for students. Chapters also reference a broad range of experts (discussed below), which further validates breadth and depth of this growing scholarship.

Given the nature of my own interests, I’ll focus primarily on the chapters dissecting the medium, Chapters 6 and 7. These chapters on formal analysis largely expand and refine Duncan’s earlier papers on “Comic Book Communication.” The theories stick largely to the expected status quo of theory: nothing overly radical or surprising jumps out, which is perhaps to be expected from a textbook.

In some ways though, this is a detriment. Despite growing works on formal aspects of the “comic medium”, most of these chapters rely on concepts inspired by Eisner and McCloud, supplemented by Groensteen and scattered others. However, much of these ideas go by with little regard for debates to their legitimacy. For example, “closure” is assumed to be true and never questioned as being valid at all (though multiple interpretations are presented). Others include the (erroneous) belief that the “gutter” somehow contributes to meaning, and the idea that each panel must be connected with every other panel in comprehension (can you say “working memory overload”?).

The primary focus of these chapters describe aspects of meaning-making, providing summaries of overview notions that intertwine across numerous levels of comprehension (sequence, layout, etc). The chapters are chock full of information, much of it useful to a beginner and some likely useful to more advanced students. I particularly liked the idea of an interplay between the reduction and expansion of information in the medium as a nice simple way to describe the status quo of considerations about the medium.

However, on the whole, numerous broad theoretical concepts are discussed without much real theoretical grist to them. Again, this might not be bad since the format is a textbook — elaboration on numerous topics would be impossible for the space.

Nevertheless, some aspects describing theories are a bit roughshot — for example describing “icon/index/symbol signs” could have benefited from better explanations, and proper terminology (it should either drop “signs” or be “iconic/indexical/symbolic signs”) and attribution to their originator (Charles Sanders Peirce) would at make for a good mention that these theories are roughly 100 years old.

Additionally, while McCloud and, at least somewhat, Eisner, are recognized for their theoretical insights and contribution to the canon, the realms of theory and praxis get blurred further in the chapters with quotes from numerous comic creators and an unheard of “comics art collector,” which among the few experts seems curious as legitimate sources. It also brings into question just what and who these chapters are aiming at: Theory? Analysis? Praxis?

Despite its limitations, The Power of Comics marks an accurate state of the field (whatever it might be) for studying comics. For good or bad, the theories in Chapters 6 and 7 reflect a particular paradigm of thinking about the medium. While it is my personal belief that formal theories have moved into a more sophisticated state, the views expressed in this book reflect what will someday be viewed as a nascent growth stage of considering the medium. For that, it almost seems like a “living history”, saying where we’ve been while knowing bigger things have and will appear.

Overall though, the book — including the theory chapters — is reasonably good for a “first textbook on comics,” and I would imagine it will fast become a standard text for those sorts of classes.

Comics reading: Competence and performance

Often times when I give talks, especially concerning layout or the visual grammar of sequences of images, one of the questions inevitably says something along the lines of "But, there's no guarantee that the reader will view a page in the proper order." The high variability of possible choices or readings of a comic page makes it hard for them to accept a steadfast theory of comprehension.

However, a similar issue was at play in linguistics back in the 1950s, and was one that Noam Chomsky importantly addressed in his distinction between competence and performance. Competence refers to the (idealized) organization of rules and constraints in our minds that guides us to understand language. Performance is the vast variability that happens in real life exchanges.

For example, someone might say something like this over the phone:

I ...uh... I went *cough cough* to the store *STATIC***--oday and, like, ...um... saw *CAR HORN*--ohn from my class in the check-*hiccup*-out line.

There are lots of interruptions, unclear portions and distractions. However, most likely a listener would glean from this a sentence like:

I went to the store today and saw John from my class in the checkout line.

The rules in your head are not bothered by the messiness of the context — your attentional system can filter out a lot of it.

The same is true of reading a comic page. Let's say you start in one panel and go to another, then realize it shouldn't have gone next. You're not belying the mental rules that go into comprehension — in fact, those rules are what tell you it's the wrong order. These actual rules of comprehension are unconscious to your awareness.

Your (unconscious) competence wins out over the messiness and variability of performance.

This same issue may be at play with comparisons of comics to film. Yes, film and comics are presented differently (one static, one moving), but that doesn't necessarily mean that their comprehension in people's minds is entirely different. The difference in presentation may be a "performance" issue, while the comprehension is a "competence" issue. (Though, in my mind there is bound to be at least some variance due to that presentation difference — motion vs. static — which will need experiments to explore... yay science!).

I should point out also that, in linguistics, there are some debates over the complete reality of this split in notions. For example, for a long time it was argued that words like "um" and "uh" are just performance clutter. However, research has shown that these actually hold meaning for the discourse (essentially signaling how long a pause the speaker is going to make before continuing to talk).

Nevertheless, for many issues facing the comprehension of "comics" (and/or film), it is an important split to make.

Abstract Comics and Visual Language

Abstract Comics is a new collection of non-representational comics by a variety of authors, including my esteemed blogging colleague Derik Badman. Besides being a beautifully done work of artistry and imagination, among particular crowds it spurs the question "If these are comics, then what 'are comics'?"

To this end, the book (and the nature of the contents in general) makes a good conversation piece. They also make a great example of how to distinguish the difference between "Comics" and my notion of "Visual Language", which is made clearest by teasing out just what parts of cognition these comics engage.

First, we should ask the question "Are abstract comics of this sort 'comics' at all?" I would have to say "YES" — simply because they call them so. However, they are not instances of visual language.

To repeat my theory... Visual language is a system of patterns (from people's heads) in that expresses concepts through the graphic modality using sequential images. So, visual language uses three interlocking cognitive systems:

1. Graphic modality
2. Meaning
3. Sequential structure (i.e. grammar)

Like spoken or signed languages, this system is culturally relative, meaning that different cultures use different visual languages (for example, "standard" manga style versus "standard" superhero style dialects), and this system is used socioculturally in comics paired along with written languages.

Comics are written in visual language (± text) the same way that novels are written in English. Novels aren't English, and Comics aren't visual language. (This equation would essentially be McCloud's position, that comics "are" sequential images).

Given this, abstract comics most definitely are comics — because they call themselves comics, they are formatted like comics, they are made by people affiliated with comics, sold in comic stores, etc. They satisfy most all the sociocultural aspects that one would expect comics to fulfill.

However, they do not use visual language. They don't use representational depictions that reflect patterns in people's heads. They don't seem to have any sort of grammar of narrative structure. They don't depict any meanings at all. In other words, they use just one of visual language's structures from our cognitive system*:

1. Graphic modality
2. Meaning
3. Sequential structure (i.e. grammar)**

They are merely playing with the graphic modality in a sequential way that entirely lacks meaning (in the conceptual sense, not necessarily the "artsy" sense).

As a result, abstract comics make a great example of comics (and art) that lack visual language. This is the inverse of something like an airplane safety card, which is representational, but lies outside the sociocultural category of calling it "comics." I should say also, that this isn't a bad thing — it's quite fun, clever, and creative, and further goes to my point that these notions of "comics" and "visual language" are separate.




* Interestingly, the one cognitive structure they do use though, is visual language's navigational system for how to move through a page layout. In many ways this is an ancillary system to the primary VL system for expressing meanings in sequence, but it is curious that this seems to be one of the formal ways in which abstract comics get to call themselves "comics".

** I haven't analyzed the pieces in the book enough, but it is not inconceivable that graphic sequences with no meaning could still retain a narrative grammar. For example, Action Star substitution incorporated into a sequence with purely visual surrounding panels could lend to this result.

Memory, Experience, and Comics Comprehension

In my last post, I discussed some traits of this quote by Chris Ware found from this blog post

“I don't like to think of my work as 'cinematic.' A movie is passive -- you're watching it, taking it in. Where a comic strip, it's completely active: you have to read it, search it for meaning, for the connection with your entire experience and your memory. Yes, you do have the illusion of watching something happen in a comic strip -- but if it's done well, it comes alive on the page like a novel. A novel is the most interactive thing ever created.”

The other thing I find interesting about this quite is that I have a hard time believing that people "imagine" things while reading comics that connect with their "entire experience" and "memory." There are two things that this quote implies:

1) That people are converting their reading experience into consciously clear interpretations (imagery, sounds, etc) while reading a comic (a notion that echoes McCloud's Closure).

2) That people's creation of meaning is entirely based on experience ("Empiricism").

Concerning the first point, I know when I read a comic, I don't necessarily feel like I "fill in" any missing imagery with mental imagery of my own. I don't visualize anything that isn't in the pages. I do understand it, and make the mental connections between and across images/words, but there is no additional imagery added. Novels do create this imagery (for some but not all people) because it isn't provided already.

This blog post has replied to my earlier posting expressing that Ware's meaning of "active" comprehension relates to this sort of filling in of sensory information that's missing. Again, I am hesitant to accept that people are actually imagining sounds, smells, motion, etc. while reading a comic.

Novels certainly allow people to create visual imagery — but vision is our primary sensory modality, so I find it unsurprising that this would happen. I'm less confident about the other senses.

SO....If you actually do feel like you create mental imagery while reading comics, I want to hear about it in the comments please!

On the second point, there is quite a lot of evidence that our understanding of meaning does not necessarily come from experience (and certainly not conscious experience). That's not to say all of it is innate, but there's a give and take between innate meaning and acquired meaning — the debate is over the percentages.

What I'd be more confident stating though is that when reading a comic, I doubt people are actively referencing overt memories or experiences in order to comprehend a sequence. Rather, they are drawing upon their abstract concepts — just like when they read a book, or yes, see a movie.

How active is comic comprehension versus film?

Dash Shaw writes an interesting post delving into the "cinematic" nature of comics that explores thoughts from authors like Chris Ware with many insightful quotes.

Relevant to some of this discussion might be that some believe comics to have predated the film techniques. Or, the idea that this is a competence versus performance issue — that film uses the same mental structures as comics, just with a different presentation (this will be a topic of an upcoming post).

Indeed, in several experiments of mine I show comic panels one after another, one at a time where the participants have no control over the pacing. My participants have no difficulty understanding these or accepting them "as comics" (no one has ever questioned the labeling).

Most interesting though is this quote of Ware's from the post:

“I don't like to think of my work as 'cinematic.' A movie is passive -- you're watching it, taking it in. Where a comic strip, it's completely active: you have to read it, search it for meaning, for the connection with your entire experience and your memory. Yes, you do have the illusion of watching something happen in a comic strip -- but if it's done well, it comes alive on the page like a novel. A novel is the most interactive thing ever created.”

I have a lot of responses to this quote, but I'll save some for a later post. Right now, I want to question what "watching it, taking it in" means with regard to film comprehension that's different than the "active" comprehension of comics. This is a common thread in comparisions, so I wonder whether Ware (and many others who also do it) is conflating the presentation of a comic/film versus its comprehension.

Is the sense that film is "less active" because it's pace of viewing is not controlled by the viewer? This to me seems like a trivial thing in terms of comprehension. The process of understanding (i.e. piecing together the meaning between images, words, and/or sound) should maintain roughly the same.

If comprehension were different, we would expect grossly different results if we presented the same comic strips in different ways in an experiment (that could use any number of measures of comprehension). Let's say we had three different methods:

1) a comic page where all panels were laid out in a grid, possibly controlled so that subsequent panels only appear when a button is pressed by the reader ("self paced reading")
2) a "self-paced reading" task where only one panel is on a screen at a time
3) a presentation with no participant control, where only one panel appears on a screen at a time for a designated amount of time

Now, I would expect no significant difference in the ability of people to comprehend these different scenarios. This is all about presentation, not the content of the strips, since those could stay the same across all of these (and other) presentation methods.

#3 on this list is essentially the same type of presentation that film uses. Granted, I will wholeheartedly agree, film's use of *moving* images certainly does change comprehension. However, there still has to be meaningful connections between and across film shots (be it live-action or animated). These would be of the same "active" sort of connections that Ware describes. Indeed, you can replace film shots for panels in the above three options and probably get the same sort of comprehension as you would for static comics. So, instead of issues of presentation, the focus of questions should instead be on issues of comprehension, like:

How does the comprehension of static versus moving sequential images differ?

How does moving images within a unit (shot vs. panel) change its comprehension?

How does the use of moving across a scene (as in panning, zooming, etc.) differ in comprehension from it's static presentation in panels (or shots)?

AND... we can't really answer these questions without an adequate theory of how comprehension of sequential images works in the first place, which is essentially what my research for the past several years has been focused on.

What is "Visual Language"? Video Talk

A few weeks ago I had the pleasure of giving a talk at the University of Toronto about my theories of visual language hosted by the Knowledge Media Design Institute. They have now posted this talk online to be viewed in full (beware, it's quite long) on their website.

If anyone is really interested in just what my theory entails overall, this is definitely worth watching. It lays out the basic principles and issues for what exactly I mean by "visual language", and how that relates to "comics", "language", "art", etc.

Note: Be forewarned that the slides they show do not have their full proper animations, so might end up looking a bit more cluttered than is intended.

Consistent reading time for comic pages?

I had a link sent to me recently asking about this blog post that claims all comic pages regardless of content are read at 3.75 seconds per page. A relevant section:

I read in one of Frederik Schodt’s excellent books on manga that a study concluded that readers spend an average of 3.75 seconds on a comic page. My own observations of myself and others has led me to believe that time frame to be fairly consistent, by which I mean not dependent on the contents of a page. Unless a writer really creates an absolutely confounding monologue or an artist completely botches an integral sequence, readers do not seem to change their flipping speed for “difficult,” wordy, nor beautiful pages. This yields somewhat counter-intuitive results, in my estimation. Single panel pages, which should ostensibly be flown through, allow one image to be lingered on or “drunk in” because that one drawing is granted the full 3.75 seconds. Pages with many panels, taken to the extreme above, should require a slower, more contemplative pace. But they do not. They seem to clock at the same 3.75, meaning the eyes need to whip through these images to make it in time.

He then goes on to advocate different strategies of layout based on the idea that readers will go through it at this magic time of 3.75 seconds. Since I wrote a lengthy counter-rebuttal to this claim, I figured I ought to post it here too.

According to the science I've seen, this does not seem to be the case. The amount of time people spend on each individual panel varies based on how much information is in it, it's order in the sequence, as well as possibly size of both panel and page, and a whole page time varies definitely the way the page layout is organized.

From a very general study of my own relating to times it takes 4 panel comic strips to be read, I found each panel at an average of 1.5 seconds per panel when readers press a button to advance through panels. But, it does vary per position and narrative structure — first panels are consistently slower, panels after major events much slower. However, if you just take that average and multiply it by 4, that gives you 6 seconds for one 4 panel Peanuts strip that has no words in it.

In my last study, I found reading times varying between .6 and 1.8 seconds per panel (small times for panels that had very little information, such as blank panels or those with just action stars), with the full 6 panel strips clocking in around 6 ±2.5 seconds.

Plus, the *uncited* study that was mentioned in the blog is for manga (and if I recall correctly, Schodt also doesn't cite the actual study), which consistently 1) use slightly less panels per page (my corpus study — "Cross Cultural Space" — showed both American and Japanese books to have 5 panels per page, but manga had a lower standard deviation), and 2) use less balloons per page. Furthermore, eye-tracking studies show that fluent readers skip over far more balloons than non-fluent readers — so, less balloons means less reading time, especially for fluent readers.

The poster here then says that he finds this time to be consistent to his own experience — but you can't know such a thing from anecdotal evidence. You would have to have measures to substantiate it.

And, even if it were true that on average pages are read at a pace of 3.75 seconds — which, I imagine there is some average time out there if one were to crunch all the numbers — there is no way that we would feel the need to allot different time to different panels based on some intuitive feeling that we "want" to read each page in a specified amount of time.

Rather, the time it takes to read a page all depends on its content and the fluency of the reader.

Why Do You Make Comics write in visual language, If You Do?

Over at TalkAboutComics, Joey Manley asks "Why do people draw comics?", and makes the observation that...

A lot of people who read comics also make them. Maybe even most people.
...
Almost everybody I’ve ever met who reads comics has, at some point or another, actually made one, even if he or she never showed it to anybody.

To me, this has to do with the nature of the visual language of sequential images. Unfortunately, our failed definition of "comics" conflates the idea that "comics=sequential images", when really "comics=cultural context" and "sequential images=visual language." By this notion, what Joey is talking about is that comic readers are highly likely to create stories (comics) using this visual language.

Perhaps this should be unsurprising then, since it means this visual language works like any other language. People get exposure to it, and imitate it in order to learn and practice their fluency.

This is not a case of people "drawing comics" the way that filmmakers "make movies" or other types of "artistic" craftsmanship. Rather, this is more like the way that Americans "speak English" or Quebecois "parle en Français" because those are the languages of the communities to which they belong. Comic readers constitute the language group for visual language in America, so their "drawing comics" is simply participating in the (visual) language of their community.

Note that this also applies to the particular graphic dialect that they might partake in: those who read manga are likely to draw like manga, those that read superhero books are likely to draw that way.

Origins of narrative

A friend of mine forwarded me this facetious article about the narrative structure in films, which left me thinking about why exactly a consistent structure like the one that is found in Freytag's Triangle might be found so prevalently.

Studies on "story structure" have consistently found that people (in our culture) have better memory for stories that follow this arc than those that don't, indicating that it isn't just a vogue thing — it's cognitively advantageous. So, on the one hand, we might attribute this to an "innate" universal structure found in people's minds. This would be in contrast to the idea that there is an "archetype" floating out in the world that stories follow. Really, as with most all human behavior, there are no structures "out there" — the only place that such structures can exist is in people's minds.

Of course, the immediate comeback that might arise would be that not all cultures' stories follow this Aristotelian narrative arc. However, a simple fix around this would be that, like the patterns in languages, different cultures might externally have different narrative patterns, but the capacity of all human minds have structures that allow those diverse patterns to emerge.

For example, most all languages use Subjects, Objects, and Verbs, though they put them in differing orders. It may be the case that analogous categories exist for narrative, but that the Aristotelian narrative arc simply is one of the patterns that these categories are put into for a (i.e. our) particular culture.

Deeper though, we should think about the function of narrative in the first place. It seems implicit amongst most approaches that narrative is for telling stories. However, why might a mind have a need to tell "stories" specifically? Rather, I think stories are merely a symptom of the broader function, which would appear to be about ordering information — particularly about objects and events.

With such an organizational system in place, stories serve a cognitive purpose as a way to facilitate comprehension and memory. Entertainment and artistry for those stories is just a affective bonus.

Legitimation of "comics"

Comics scholar Domingos Isabelinho posts a critique of Thierry Gröensteen's paper on comics' search for legitimation. Domingos keys in on this quote in particular, so I'll do the same:

“Although comics have been in existence for over a century and a half, they suffer from a considerable lack of legitimacy. To those who know and love it, the art that has given us Rodolphe Töpffer and Wilhelm Busch, Hergé and Tardi, Winsor McCay and George Herriman, Barks and Gottfredson, Franquin and Moebius, Segar and Spiegelman, Gotlib and Brétecher, Crumb and Mattotti, Hugo Pratt and Alberto Breccia, not to mention The Spirit, Peanuts or Asterix… in short, comic art, has nothing left to prove.”

His critique of this is that an analogy to literature would be ridiculous: why would literature have anything to prove?

However, we can take this one step further. In his work, Gröensteen conflates the socio-cultural "comics" with "sequential images with/out text". If he means the former part of this dichotomy (comics, the sociocultural phenomena) then Domingos' argument stands. If he means the latter (the expressive system of sequential images), the claim becomes even more bizarre:

Why would any type of expressive form (drawing, writing, etc.) need legitimation? Do we really have to have justification for why drawings (in sequence) are worthy of attention at all? From a cognitive perspective, this seems crazy: all of these things are just ways in which humans express themselves.

To exemplify this, try replacing some of those names above with friends of yours, and the last sentence as: "in short, English, has nothing left to prove."

The justification of English should be self-evident, and so it should be with the visual language of sequential images too.

Review: The System of Comics by Thierry Groensteen

Groensteen, Thierry. 2007. The System of Comics. Translated by B. Beaty and N. Nguyen: University of Mississippi Press.

Printer-friendly pdf

The best thing about the new translated work The System of Comics by Thierry Groensteen is that it hopefully reflects an increase of English translations of international works on comic theory. There are numerous offerings by European, Japanese, and South American authors that rarely make their way into American scholarship, and more exchange of ideas can only be fruitful to the field. Groensteen is regarded as one of Europe’s leading historians and theorists of comics, and in its original French, the Systeme de la Bande Desinée is heralded by many as a "must-read,” so it would be a logical starting point for such a trend.

Moreover, Groensteen’s book is offered by many as a “serious” alternative Scott McCloud’s Understanding Comics, which has gained massive popularity, especially in America, yet remains dismissed in many scholarly circles since McCloud himself is not an academic. Comparisons between the two works has become commonplace (including the book’s own introduction), and this review will be no exception. For his own part, Groensteen does not engage with McCloud's theories except in a singular passing endnote, which is a shame since their ideas share many similarities.

While my French is not sufficient to assess the overall quality of translation, it certainly did not read comfortably. At times, the English felt unnatural, and word-choice often seemed clumsy if not uninformed. For instance, calling the psychological device of an "eye tracker" an "eye path follower" betrays a lack of competency and/or desire to find the accurate vocabulary (whether on the part of the translators or author is unknown). Additionally, while the endnotes were often helpful, the absence of a reference section is quite noticeable, and the endnotes do not include all the cited bibliographic information. As an “academic” book, such an oversight is a bit peculiar.

Nevertheless, the writing should only be a surface issue to the actual ideas in the book. Perhaps the most appropriate place to start is where he does: with the definition of "comics.”

Problems with definitions

Groensteen begins with what has become a common exercise in the study of comics: defining “comics” itself (no doubt bande desinée originally). He carefully deconstructs the faults of various definitions that have been proposed by various realms of scholarship. He rightly shows it cannot be guided by a single "essence" like text/image interactions, and decries a definition of comics as episodic narratives, among others.

Rather, at the outset, he identifies "comics as a language" a “system” that arises out of the "combination of a… collection of codes,” most strongly motivated by “iconic solidarity” — a fancy term for the contribution of several images. While somewhat more flexible, this emphasis on the visuals sounds a great deal like McCloud’s more rigid “juxtaposed sequential images,” but lacks a reference or engagement with his ideas despite discussing many other scholars works who Groensteen disagrees with.

Truly, in its similarity, Groensteen’s definition falls into the same formalist trap as McCloud's in failing to separate the structural notions of creating images/writing from the socio-cultural role of “comics” as objects/artifacts. This habit may have been inherited by them both from Kunzle’s influential The Early Comic Strip, whose recasting of the term “comics” on pre-1800s sequential images engendered many (like McCloud) to seek out a definition to include all possible historical examples. However, this goal is a red-herring, and as Horrock’s points out, there is no reason for sequential images from diverse historical contexts to be bound to the same socio-cultural context of the contemporary notion of “comics.”

Indeed, "comics" as a social artifact refers to numerous qualities, including 1) physical objects (strips and books), 2) a collection of genres, 3) an industry, 4) a culture/community, and others that are all tied to a context of the modern era. On the other hand, sequential images do create a language: a “visual language” that combines with text to be used within those social objects called "comics." "Comics" are not this visual language. "Comics" are a social object written in a visual language that combines with text. If novels or magazines are written in English, why should “comics” be a language, instead of be written in a language?

While Groensteen strives to strike out an abstract notion for his definition, he frequently reinforces the conflation of "comics as a medium” with "comics as a social artifact” — particularly a physical object. He writes that comics owe their birth to the technological development of lithography (8). Further on, his analysis of both word balloons (69-75), panels as “measurable in square centimeters” (28), and the expressive power of sequential images through “arthrology” rest wholly on the physicality of and across pages. However, physical objects cannot be languages — especially in the abstract sense of them being a “code” — and his reinforcement of it runs contrary to his own definition of “comics” that claims to guide the broader work.

The "System"

As invoked by the title, Groensteen’s work seeks to sketch a “system” by which “comics” operate. To his credit, he carefully moves in analysis through varying levels of interactions in comics’ structure, accounting for most of its forms. He eschews the method of previous structuralist approaches to treating the form as a “language,” which aim to dissect the medium into its minimal units, instead aiming for its broader “articulation” — larger levels of structure. However, Groensteen’s grand system is little more than an extensive taxonomy with terms that essentially mean “the principle by which (pick your taxonomic portion) operates.”

Groensteen begins by discussing the “spatio-topical system,” the various spatial elements at play in comic pages. He first notes the “hyperframe” as the delineated space of the page, in contrast to the “multiframe” — meaning the relation of all frames that constitute a comic piece, including “the sum of the hyperframes.” This chapter also includes discussions of the functions of a panel, usefulness of margins, positioning of balloons, and various facets of page layout, such as inset panels, the “strip” of a tier of panels, and possible functional roles for types of layouts.

Now, it's not that I think Groensteen's theory of comics is invalid or wrong, but that it is uninteresting.

While at times insightful, he labors through discussing details of relatively lackluster observations, such as the various ways in which a balloon visually interacts with a panel border. It is understandable why a taxonomic distinction can be made between balloons that touch the panel border and those that overlap it, but does it really add substantially to knowledge about the function and understanding of balloons as a graphic/narrative element on its own?

If this discussion led to a substantial observation about the constraints that this interaction creates on such a relationship, this breakdown would seem more significant (for instance, that two balloons cannot cross tails such that their speakers are in opposite panels than the balloons). However, no revelation of this sort is reached, and most of his analysis focuses solely on the physical aspects of the relations of balloons to panels. Such is the features of most of his discussions. All of this leaves the question of what this analysis offers a theory of comics understanding except for surface descriptions of a (fairly banal) phenomenon?

The next two sections are devoted to the principle of “arthrology” to describe either the linear relations of panels to each other (restrained arthrology) or the relations that one panel might have to others in a non-linear sense (general arthrology). Unlike McCloud’s panel transitions, Groensteen does well to recognize that panels make connections beyond their immediately juxtaposed neighbors, yet does not give any hint as to how. McCloud’s transitions at least attempted to characterize relationships between panels, which is largely why the approach is so appealing, but Groensteen leaves such detail aside, preferring instead for gross scale abstract principles. He describes “braiding” as the principle guiding arthrology — essentially the function of making connections across the multiframe.

On the surface braiding and arthrology appear to be about the creation of meaning, like McCloud’s “closure,” or perhaps a comic version of Halliday and Hasan’s (1976) classic notions of local and global discourse coherence, which Saraceni (2000) also attempted to map to comics’ analysis. However, Groensteen does not even come close to talking about semantic concerns in this way — despite purporting to (the back cover describes the book as “An authoritative exploration of how the comics achieve meaning, form, and function.”).

Really, braiding and arthrology are a theory of compositional relationships. What Groensteen focuses on is not in any way a system of how the content of panels lends towards making meaning — it is primarily about the relationship of the visual composition within a panel to others on a page and other panels scattered throughout the broader work. In particular, his theory observes the recurrence of compositional or thematic similarities (such as motifs) across varying distances of space, be it a page or a whole work, and whether the position in layout of those panels has significance (such as in the first or last position of a page).

While these observations provide a very different perspective on comics pages, they do not even scratch the surface of the “holy grail” of questions about comic sequences that they appear to address: how do sequences of images create meaning? Indeed, Groensteen dismisses such aspects as merely about "storytelling."

Rather, arthrology, like the other aspects of Groensteen’s system, once again highlights a physical feature of comic pages in what is proposed as an abstract code of understanding. The theory treats the medium as an “art object” to be analyzed like a multi-canvas painting, as opposed to a communicative medium in the McCloudian sense. Indeed, such an approach would be akin in verbal language to recognizing that various patterns of sound appear throughout different words in a story — perhaps telling you something about phonetics, but nothing about meaning. All told, Groensteen’s system limits itself to only describing the surface aspects of the medium, without pushing towards any deeper constraining principles.

Despite the criticism of the workings of his system, Groensteen is nevertheless an astute observer of various components of the comics form, and his numerous insights on comic elements betray a deep commitment to dissecting the medium. The System of Comics does reveal gems of this intuition at times when discussing various components, though the overall theoretical architecture in which they are embedded is not commiserate with the value of their insight.

Scholarship and Paradigm Shifts

The most troubling thing I found about The System of Comics is the overall orientation to scholarship that this work represents. Throughout, Groensteen’s writing conveys an attitude as if the theories are entitled to be significant, echoed in the introduction where Beaty and Nguyen hail it for emerging from the rich semiotics tradition, as if that inherently legitimates its ideas. This is a direct knock to McCloud, who they note has “been criticized for…lack of theoretical sophistication” (1). Though, this criticism has only come from academics unfortunately perpetuating the stereotype of ivory tower snobbery, as if scholarship must be done in the academy. It is hypocritical at best for any scholar to deride McCloud for not engaging a broader literature while lauding a book with very similar ideas to McCloud’s without any citation or discussion of them. So much for the value of “engaging the literature.”

Truly, Groensteen is the anti-McCloud, keeping the keys to theory locked away in the ivory tower, reachable by only those willing to slog through the exposition to reach it. While McCloud’s work about comics is presented in its visual language, there is some irony that The System of Comics takes “iconic solidarity” of images as its crux, yet is almost devoid of graphic examples. Perhaps this is why McCloud is ridiculed so much by Groensteen-minded theorists: he willingly gives away the goods to the rabble without a fight.

If McCloud lies on one extreme of being too accessible (as if that’s a bad thing), Groensteen’s work is the inverse, reflecting the worst of academic jargon and inapplicability of “theoretical sophistication” — to the extent that the terminology obfuscates the actual theories (to academics and layfolk alike). Groensteen offers complicated names and lengthy descriptions to what are otherwise fairly facile observations about surface phenomena.

Moreover, numerous questions are left unanswered, for instance how this theory is useful or applicable to 1) describing how this medium of sequential images communicates, 2) contrasting various comics’ structure with each other, or 3) describing the relationship of the visual language in comics to other modes of human expression?

Groensteen’s “system” accomplishes none of these. These are all questions that are important to semiotics as a discipline, so why is Groensteen's approach unable to even broach them? It only states vague principles for nearly obvious observations, with fancy names and a semiotic tradition to validate its claims. It is scholarly hand-waving at its best, and a reflection of why the invocation of “semiotics” garners more eye-rolls than awe in many circles.

In Kuhn’s renowned discussion of paradigm shifts, he describes that prior to a major shift many similar theories will emerge in competition with each other, yet all pointing towards similar intuitions. In this case, Groensteen taps into the common intuition that the system found in comics is somehow similar to the system of language. Indeed, he begins with a strong statement about how comics are a language, yet his analysis paints a picture of the comic medium that is decidedly un-language-like.

While his stated focus is not on “minimal units” — thereby avoiding topics such as how the system is understood as a graphic domain and meaning-making for signs and symbols — he states nothing resembling a grammar for how the sequence creates meaning (though claiming that is his intent). If his aim is to describe a “code” that shows this is a language, one would think it would involve structures akin to language. All this leaves doubt as to whether Groensteen would actually know what being “a language” would entail in the first place (aside from metaphoric extension).

To this extant, Groensteen’s total work certainly marks a valiant attempt that can take its place next to other approaches that reflect the growing pains of a discipline, sharing the intuition that the comic form should be compared to language (including: Eisner 1985; McCloud 1993; Saraceni 2000; Cohn 2003, etc.). However, this piece and its theories are not the revolutionary new paradigm for comics that will sweep away all others, and will likely follow the path of the branch of semiotics from which it hails — being considered largely passé in the study of language.

References

Cohn, Neil. 2003. Early Writings on Visual Language. Carlsbad, CA: Emaki Productions.
Eisner, Will. 1985. Comics & Sequential Art. Florida: Poorhouse Press.
Halliday, M. A. K., and Ruqaiya Hasan. 1976. Cohesion in English. London: Longman.
McCloud, Scott. 1993. Understanding Comics: The Invisible Art. New York, NY: Harper Collins.
Saraceni, Mario. 2000. Language Beyond Language: Comics as Verbo-Visual Texts. Dissertation, Applied Linguistics, University of Nottingham.

Comics and the Brain... almost

Nagai, Masayoshi, Nobutaka Endo, and Kumada Takatsune. 2007. "Measuring Brain Activities Related to Understanding Using near-Infrared Spectroscopy (Nirs)." In Human Interface and the Management of Information. Methods, Techniques and Tools in Information Design, 884-93. Heidelberg: Springer Berlin

Looks like I was beat to the punch... I've found a study from last year that analyzes the activity in the brain while reading comics. However, it doesn't say much.

The authors use near-infrared spectroscopy to measure blood flow in the brain while reading comics. This technique uses infrared light to measure where blood flows in the brain, which can thus indicate the brain regions involved in various behaviors. They find that "the left prefrontal lobe region is activated when people actively try to understand the comic stories and to memorize their contents for reporting in the future."

However, there are extensive problems with this study. First, the number of stimuli they use is extremely small (only 6 strips) as is their population (13 people... which does not add up to counterbalancing). Comparatively, the study I'm planning will use 180 stimuli per trial (720 strips total) and use somewhere from 24 to 36 people.

Additionally, the increase in blood flow that they observe only occurs in "reported" conditions — where subjects are actively making a judgement about the stimuli, as opposed to scenarios when they are not. This seems more to reflect the well-reported cognitive activation for making judgements than anything about the structure of the comics themselves.

So... this really doesn't tell us much about comics and the brain, but its nice to see other people are at least taking stabs at it as well.

Collaborative drawing

Last weekend on public television I saw a fantastic biography about Pete Seeger, the influential folk singer and activist. Throughout, Seeger stressed his desire to sing with people, not to people — motivating music as a collaborative endeavor. This sentiment is echoed in the accessible book, This is Your Brain on Music, which points out that music as "performance" by people on a stage to other people seems to be a fairly new thing. Traditionally, music was a group activity that was not reserved for those of express "skill" and training.

Drawing is much the same way. We often make a huge break between those with or without "talent" — resigning people to the misperception that they "can't draw", when really our biological endowment ensures that we all can draw. Really what is at issue is a level of fluency, and most people just don't develop with the proper exposure or practice.

Language, like this sense of music, is entirely collaborative. And, it is learned collaboratively, unlike most learning of drawing. In some cases, drawing might be instructed, often very well, though this is far from simply being interactive in the sense that you learn just by participatory immersion.

On a productive sense, drawing also is highly non-collaborative in our modern life. Belonging to a print-culture, most drawers and readers are separated by huge distances of space and time. This isn't always the case though. Sand narratives by native communities in Australia are highly interactive, drawn in real-time communication.

Humans are an intensely social animal, and my gut tells me that nearly all of our expressive capacities developed and thrive in such collaborative interactions. The question is: how in our modern ecology can we facilitate such usage for visual language? Will we have to rely on technological breakthroughs (ex. digital whiteboards), or can it grow organically without the crutch of engineering?


For those interested in more about this, my article from a few years ago "Interactive Comics" probed a lot of these ideas.

Essay origins

So far I've been very pleased at the response to my latest essay, "Navigating Comics", on how people navigate through page layouts (pdf). As several responses have been rolling in via email and elsewhere, I intend to do a post soon addressing concerns in that feedback. However, I think it'd be informative to first talk about the origins of this paper.

Back in 2003 when I was drawing our political book We the People, every now and then my editors would tell me they had trouble knowing exactly where to go in the sequence. Often this happened in consistent situations (like what I call "blockage" in the paper).

Most of the times I'd either simplify the layouts or make some graphic fix (like a trail) to indicate a clearer path. However, it got me thinking... My editors were quite a bit older than I was, and weren't all that experienced comic readers, so I wondered if this lack of experience mattered in their reading habits? (or if I was just needlessly making things difficult)

So, I designed this study to test that. I had a booth at ComicCon 2004 that year to promote the book and my other works, so I designed a simple pamphlet people could make responses in and tested people throughout the convention.

I could tell immediately that the results would be interesting, I just had to wait another three years to learn the statistics necessary to show them (d'oh!). The theory with the tree structures predated the experiment by at least a year, but it didn't really say much without knowing about people's actual preferences. It's exciting to see that my suspicions for creating the experiment were borne out in data.

Every now and then I get a response to my work along the lines of "Why do theory? Why not do something related to praxis?" While theory can be interesting, enlightening, and much of science is simply about discovery without practical applications in mind (ex: penicillin), another reason is that theory can sometimes wrap back around on praxis. I like to believe that this is one of those cases, especially given that it came from those origins.

Art, Language, and "Cognitive Equivalence"

When I usually speak about the Art versus Language Perspectives, I usually couch it in a view that there are "different potential ways our society treats graphic images." As I just realized, stating it in this way maybe obscures the true intent of the distinctions.

Really, this is a hypothesis about cognition.

At the heart of my theory of visual language is the observation that we have three modalities by which we can convey our concepts: Sound, Body motions (faces/hands), and creating graphic images. That's it. To push it further, the theory is that when these modalities take on structured sequences governed by a system of constraints (a grammar), it becomes a type of language: speech, sign language, or visual language.

The "Language Perspective" assumes that all systems of conceptual expression work in similar ways — what we can maybe call "cognitive modality equivalence" or some such. Under such a view, we would expect for the graphic modality (drawings) to operate under the same principles as the verbal and manual domains.

If you look at the ways that speech and gestures (and sign language) grow developmentally in children and are used and treated in society, you see certain patterns — conventionality, imitation, communality, etc. While many of those patterns do emerge in the graphic domain, they appear "dampened", are dismissed, or just aren't recognized as such.

So, the question becomes raised: "Why don't you see these things fully in the graphic modality?" and/or "why don't we know them when we see them?"

The offered explanation is the Art Perspective — a cultural force that suppresses the patterns that would normally emerge from any other modality of conceptual expression. With polar opposite emphases, the Art Perspective works to dampen the "usual" course of development and treatment for graphic images.

So, given this, a new set of questions can be asked about this underlying "cognitive modality equivalence": What are the trends that a conceptually expressive system shows in development and society? How do modalities differ? Do these trends reflect broader cognitive processes than just conceptually expressing systems?

Coherence-building in comics

Bridgeman, Teresa. 2004. Keeping an eye on things: attention, tracking, and coherence-building. Belphégor 4 (1).

Bridgeman's article discusses various aspects of coherence-building throughout comics structure — "coherence" being the discourse theory notion of a salience across various units. It thus joins various other works that apply discourse theory to comics, though dabbles in cognitive science a bit as well.

The piece covers a lot of ground over various parts of comic structure (style, color, composition, layout, etc), and it uses cognitive principles to at least elucidate the elements of structure fairly well, though it sticks to a fairly generalized notion of them. On the whole though, not much is "new" about the work presented here. It takes the fairly overt elements of structure and simply maps cognitive-theory-lite to them, while also drawing from a well-done mixture of McCloud and Groensteen's ideas.

McCloud and Groensteen's theoretical orientations are often put at odds with each other, yet this paper makes ample use of both of their theories. While I may not particularly subscribe to either of their theories (I do have my own ideas, you know), it's at least nice to see that not everybody falls into one camp or the other.

Partly though, the non-novel nature of the paper may be due to "intent," which is less to provide a cognitive analysis of the structure, so much as (it seems) to use cognitive principles for analysis. This seems to be an inherent disciplinary tension though. While I do think it succeeds as an application of cognitive theory to literary analysis (which most of the paper is devoted to), I'm also wary for whether it knows the difference between the two intentions.

Podcast on Visual Language

Later this month I'll be giving a talk at the VizThink Conference in San Francisco, so to promote it I did a podcast yesterday. Here's the original post, though they were nice enough to give me the feed here as well.

Mainly I discuss the Big Ideas behind my visual language theories. What is "visual language"? How does it relate to "comics"? Etc.

Enjoy!

Visual language in society

As I've mentioned before, I think that the meshing of the notion of "comics" with sequential images (a simple glossing of what I call visual language) actually hurts the perception of the graphic form by pigeon-holing it into a specific cultural context.

If this form of communication was actually used like a language there would be no reason people would call it "comics," and I certainly wouldn't have to be arguing that it is literally a language. Its recognition as a language would be self-evident from its usage.

This is why I always correct people who claim that we have a visual culture, or that people these days have a vast visual literacy. They have a familiarity with technological or cinematic representations, certainly. But, when 2/3 of America can likely not draw a coherent narrative sequence? Visual fluency, I doubt they have.

Truly, the need to argue for a "visual language" only comes out of a society where such usage is exceptional, not the norm. You can't have a culture where people claim visual communication is vast and prevalent, while at the same time have books arguing for increased usage of it.

Can you imagine books arguing for why people should be using language more? It wouldn't happen, because language is so prevalent and pervasive in society that to do so would be boring.

That's the extant of visual language I'd like to see — where someday people will look back on my writings and think its bizarre that someone had to argue for someone to even need to make the argument that its language.

I'll show you my paradigm if you show me yours

I'm shocked at the response that my quote in yesterday's ¡Journalista! (thanks Dirk!) has generated. Is it really so hard to believe there's a separation between the structure of a system of expression and how it's used in "literary" contexts?

Ok... maybe I'm not so shocked...

For spoken language, this is simply the difference between studies in lingusitics/cognitive science and literary studies. Not seeing that split for "comics" is one of the issues wrapped up in the muddled understanding of graphics and "comics" in our society.

And, I should say, I don't believe that literary and linguistics/cognitive analysis don't or can't complement each other, but recognizing the division is important at the outset. One is involved with idenitifying cognitive processes and patterns of usage and behavior. The other looks at how those patterns are used to create some sort of expression (and possibly, larger level patterns). One discusses meaning for the base semantic understandings of cognition. The other discusses meaning layered on top of those cognitive processes (often "interpretive").

The structural analysis lends itself to informing the literary quite easily. If I were to propose a method for categorizing panels based on how many "characters" they contain (which I have), even as a structural analysis, literary works can use it to discuss the sorts of concerns they have. This doesn't work as well in reverse — categorizing panels (or layouts, as Peeters did) as "decorative" versus "rhetorical" does not offer me a way to study processing or structure (though I'm open to being proven wrong). However, it does work within the contexts of literary analysis.

Neither approach is inherently better or worse within the contexts of its own intents. However, the motivating factors behind the paradigms need to be recognized as very different. To this end, the label "comic theory" is being used in very disparate ways that do not necessarily inform each other. While I've found many who go ga-ga over it, I find French "comic theory" as largely unusable and uninformative to my conception of "comic theory"—because it is of a totally different paradigm. (...and also why, I'm guessing, that they tend to dismiss McCloud's work, which I find to be far more interesting than theirs)

My point is, that like the divisions between any paradigms, it's not so much the answers being provided that are different, but also the questions. The clearer this can be made, the more there is potential growth for "progress" on all sides.

Review: The Language of Comics by Mario Saraceni

Saraceni, Mario. 2003. The Language of Comics. New York, NY: Routeledge.

Saraceni's The Language of Comics is one of the few books that attempts to present a holistic theory of how comics work, and draws upon work from "applied linguistics" no less. The book is actually a stripped down version of his dissertation (as is his article "Relatedness" from the Graphic Novel collection). (And good luck finding the dissertation... I had to print it from microfilm on interlibrary loan).

Unfortunately, much of his approach seems to feel of grafting McCloud's work to ideas in applied linguistics in a simplistic (and uncited) way. For instance, he proposes a gradation between semiotic types (like symbols and icons), and can well be compared to McCloud’s Big Triangle.

He treats the sequential aspect of panels as equal to sentences, giving them a “discourse theory” type analysis (like the dissertation by Stainbrook). Saraceni claims meaning is created through commonality between elements in panels, alternating with successive new and given information. He also uses "semantic fields" (connected meanings: like how "snow, caroling, pine trees, and presents" invokes "Christmas") to unite panels not encompassed by this information structure.

However, in doing so, he eschews the role of linear sequentiality, yet provides no argument for why people do indeed read in consistent sequences. The result is essentially a watered down version of McCloud's closure — which it is: his dissertation has the theory in full, and exactly does shoehorn discourse theory onto McCloud's transitions. Some of his insights here are useful and enlightening, yet they deal entirely with "exceptions." He rarely discusses "run of the mill" things like the depictions of events, instead culling his examples from very experimental comics storytelling, like Peter Kuper's The System.

Other chapters cover things like word balloons (perceived as equivalents to direct quotes) and drawings of eyes to buttress a discussion of subjective and objective viewpoints. The final chapter is about computers, which seems out of the blue and has next to nothing to do with comics.

Since it uses applied linguistics, much of the book feels attuned to what might be useful for literary studies. To this extant it might work very well. However, as a theory of "meaning" it falls short, largely because it does not address any type of cognitive system, and lacks even McCloud's precision of surface categorization.

My biggest gripe about the book is that it is presented as an introductory textbook as part of the Intertext textbook series, has no citations outside a "recommended reading" list in the end, and is written with a matter of fact tone that presents it as an authoritative stance on the body of knowledge of this field. The truth is that no such body of knowledge exists at this point for "comics theory." Right now, we're in that period of science where lots of different viewpoints are popping up, just waiting for an encompassing paradigm shift to sweep in and take over.

Even if I were to come out with a book of my full theories, it wouldn't be a de facto textbook because it would be my views drawing upon that body of research. As a result, to those "in the know," the format and style make this book seem misleading in its intents for fronting Saraceni's views as well established scholarship.

To end on a good note: Though I think it fails in not using a cognitive approach, I do like that the book tries to use concepts from linguistics with "comics." It shows that this type of approach is not just intuitive to me, but to others as well, and locating the book in the broader field of linguisics is good for the field as a whole.

16 panels that are Still Conventional whether you think they work or not

I neglected posting this ealier in the week, but John Morris has a parody/homage to Wally Wood's 22 Panels that always work over at Comixpedia entitled 16 Panels That I Don't Think Work All That Well. There are a couple things I find theoretically interesting about it.

First off, it is a great compilation of conventionalized patterns used in many comics. Just like WW's 22 Panels and the Peanuts patterns I've been finding recently, this list excellently shows how there are systematic and conventionalized patterns in the visual language used in comics. This is in contrast to the view that graphic creation is unrestrained because it mimics perception, and thereby lacks an experss system of mentally stored patterns. Astute observations like these 16 panels excellently show that there is systematic and patterned visual vocabulary used by "visual language speakers" (and I wish more people would do work like this!).

The second thing this list shows is a preference for the use of some representations over others (WW's 22 Panels does this too, though positively as opposed to negatively). Linguistics has often been in perpetual debate with journalists/English teachers/etc. that believe there is a right or wrong way to use language. We are often told "not to end a sentence with a preposition," or "not to split infinitives" — though these are not in any way real rules of English grammar. (They were prescribed by grammar book writers in the 18th century who were enamored with Latin — so they advocated Latin's rules for English, not at all being sensitive to the fact that, y'know, they're totally different languages!!).

This list's intent is prescriptive in the same way. Despite these being consistent conventional trends used in this visual language, they are gauged by their value in usage. An additional aspect to this is the Art perspective most invoked in the comments below the article. Most people object to these panels simply because they are conventional! They're "overused," which means they aren't new and innovative/original — which makes them undesirable to the Art viewpoint.

However, none of this mitigates the fact that these panel types are conventional. The linguist would say "they're all part of language, let's observe how people use them" while the prescriptivist says "they're part of language, but they really shouldn't be, and those who use them are less sophisticated speakers."

It's interesting to note also that no matter how loudly prescriptivism might object to such "bad" usage, it never has an effect on shaping language usage. It's not like split infinitives have gone away because people advocate against them! Nor do I suspect these 16 panels to go away either.

In some ways both aspects of a list like this shows some good headway in recognition of this visual language as a language on the whole. Not only are people recognizing the patterns, they're also judging them prescriptively, just like any other language!

Visual Fluency... aka "Pissing off Writers"

There's a been a little hullabaloo lately about Jesse Hamm's recent blog post on "Why Comic Book Writers Oughta Mind Their Own Business," and the Mark Waid reply. Attitudes displayed by both sides aside, I actually agree with Jesse's sentiment, but I think there's some deeper issues going on here in this "artist vs. writer" clash.

What's really at issue here is the level of fluency a person has in producing this visual language that is used in comics. Most of the time graphic production is viewed as a matter of skill — how good someone is at "drawing pictures" or maybe "telling a story" as opposed to being a fluent vs. not fluent producer in an explicit system of expression.

"Artists" — those who are demonstrably graphically fluent — rightfully are justified in feeling perturbed at being dictated to by writers, whose graphic fluency might be questionable. Jesse's post highlights the disjunct that occurs when "artists" are being directed by a writer who is not graphically fluent.

There is a difference between fluency in comprehension and production. Children whose parents speak different languages than the general populace sometimes might be able to understand what their parents say, but they might not speak it. This occurs en masse in our society for visual language — especially since it's a print culture with no reciprocal exchange of expression. Most people in our culture are fluent readers of visual language, but just being able to comprehend is not sufficient enough to be able to produce.

Beyond proudction, there are some cultures — and individuals — that cannot comprehend sequences of comic images. Reading ability between fluent comprehenders also varies as well. Experimental eye-tracking data show explicit differences in the path of eye movement between "expert" and "non-expert" comic readers when navigating through comic pages. Other studies have shown that skill in comprehension of panel sequences and memory of them increases with age, indicating a maturation of expertise.

Further experiments show that children from countries with limited comic reading (i.e. like America) cannot produce sequential images at a high proficiency (in some studies, over 2/3 of the subjects could not create sequences where one panel has some connection to its adjacent panels). In a truly fluent populace like Japan where most all children read and draw manga as part of the culture, 100% of children tested can create sequential images at a high degree of proficiency using complex visual grammar.

We need to move beyond the naive belief that drawing is just a "skill" in a "universal" "art form" (that is equally expressible by those who do or don't have the skill) and towards the cognitive reality that it requires a graphic fluency in an explicit system of expression that must be learned and developed... leaving a large group of people with degrees of limited fluency.

(... just like any other language).

You're a good grammatical construction, Charlie Brown

I've recently been thinking quite a lot about how best to start testing my theories of visual language grammar. Since I'm in a psychology program, I've got to actually think of experiments that might yield reliable and significant results (hopefully).

One of the main ideas I had was starting off using a corpus of comic strips, so I wouldn't be biasing the study with my own drawings. I hit on the thought that Peanuts strips would be perfect for this since 1) there are a ton of wordless ones, 2) they're well recognized culturally, and 3) they use a fairly simple bare bones structure with 4) nearly always with 4 panels.

So, thanks to a very kind donation from Fantagraphics, I am now pouring through several volumes of The Complete Peanuts strips in search of all the wordless/minimal text ones I can find (there are a lot!). Hopefully, by summer I should be testing peoples intutions on the grammar of these strips, and eventually looking at their brainwaves while processing them (fun!).

One of the things that has jumped out at me is how so many of the strips use systematic patterns that I haven't noticed before. Previously, I've talked about the visual grammatical pattern of the 'Set up - Beat - Punchline' construction (as coined by Neal VonFlue). This is the pattern that sets up the joke with dialogue, then has a pause panel, then ends with the punchline. Well, Schulz seems to use a few other patterns a lot as well.

The most intruiging to me is one that is almost exactly like the SBP pattern, only the "beat/pause" panel isn't actually a pause: it's an "action" panel (SAP?). Instead of a passive type "rest," the space is filled by some wordless action that sets up the payoff with the final panel punchline. I've only looked at the oldest of the collections (the 1950s) and have only seen a few actual SBP constructions. I'm curious whether or not this SAP pattern preceded/led to the SBP one.

Another pattern has the first three panels as wordless depictions of an event, only to have a final panel with a punchline that explains or comments on the actions. This one happens extremely frequently, and sometimes takes on an additional characteristic of having the first panel depicting an action as well. It starts with an event that sets up the primary event that unfolds in the rest of the panels.

Patterns like this are fun to find, but can also be challenging theoretically. At least as far as developing a model for my visual grammar, sometimes I'm hesitant of how to notate certain panels, and often debate which is more correct. Imagine not only trying how best to describe how nouns and verbs combine, but also whether or not things are nouns and verbs in the first place and/or whether those categories are appropriate at all (when there's good evidence for all).

And, unlike with homework, there is often no answer key that I can check with someone else (except, hopefully, what my experiments will reveal). I've always found this "working without a net" to be a little scary, but at the same time exciting since it portends new and uncharted territory. I suppose it's the feeling of truly doing science instead of just learning it.


Note: As long as I'm giving thanks for donations, I should also mention the kind contributions of TopShelf, Drawn & Quarterly, Top Cow, Oni Press, and Dark Horse Comics. Their generosity will make a huge difference in these visual language studies and are greatly appreciated!! If you are from another company and would like to donate to this cause, please contact me...

New Article: Loopy Framing

At long last, I have a new article up at Comixpedia called "Loopy Framing"! This one tackles the similarities between word balloons, thought bubbles, and panels, arguing that they are all essentially the same thing. Oh, and it reveals some nice aspects of human cognition along the way.

I should also note that this article wouldn't have been nearly as good looking (or timely) without the fantastic illustrations by Tim Godek. Since I'm generally swamped with schoolwork, Tim kindly agreed to help out and make this happen, to which I'm greatly appreciative. Be sure to thank him too, by checking out his site and blog (which are on my regular reading list).

I look forward to hearing your feedback, either here or on the article's thread. Enjoy!

Thoughts on a visual e/i-language

Big breath; prepare for a long post...

Most of the time, people think about languages as being "out there in the world" – consisting of a conventional list of words and rules that we all access. This leads to people commenting frequently things like "that's not a word," out of the belief that there is some external authority that dictates what is or isn't in a language.

In contrast, a cognitive approach to language looks at it as a phenomenon of individual's minds, and that the mutual conventionality of many minds creates the artiface of a system that is "out there" abstracted from those individuals. Several linguists have tried to delineate the relationship of a language to its speakers.

The first major one would have to be Ferdinand de Saussure, who made the breakdown between langue and parole. While parole describes a language in use as a dynamic social activity, langue was the notion of an abstract system of expression beyond its usage.

Noam Chomsky did Saussure one better though. Chomsky maintains that this system of expression is located squarely within a human mind/brain, what he calls an internal or i-language. In contrast, and external or e-language covers both of Saussure's terms: 1) external speech behavior and 2) the system as something in the world, abstracted as we call English or Chinese.

In reality, E-langauges are built out of the mutual intelligibility of people's i-languages, which often differ based on geography and community. A good example is an irregular derviational rule like the plural of "cactus." For a Southern Californian like me, the plural is the irregular "cacti," because it's common enough in daily life to be out of the ordinary. But for, say, a Bostonian who lives around me now and never interacts with them, the prickly stuff might be called "cactuses" using the plural "s" rule of English. Both are "right" in a cognitive sense, because the cognitive structure differs based on ecological context (most jargon is like this too).

An "e-language" distinction dislikes this, yet it's the reason that dialects exist at all. They are just degrees to which people's mutual intelligiblity of i-languages group in a graded way. The "r"-less Bostonians down the road from me certainly have different rules that they follow in their i-language than I do, but our systems are close enough that we understand each other.

And... all this is setting up some useful concepts for what I really want to talk about: drawing.

I've discussed previously that there is a cultural force of an "Art perspective" that affects the development of people's drawings skills in our culture. While they may be cognitively inclined towards imitation, the Art perspective guides them towards having an individualistic style and away from using the shared structures of a community.

OR, in Chomsky's terms... the Art perspective pushes people to have i-languages that (for the most part) don't build into an e-language. There are some exceptions to this of course, maily in conventional symbols like word balloons and speed lines.

Note the semiotic allowance here though: the symbolic aspects must be conventionalized and thus easily build to a shared structure that becomes more "out there" in the world abstracted from users. But, the broader iconicity of the system (that the signs resemble what they look like) allows for a degree of mutual intelligibility without mandated conventionality. This lets individuals' styles be inexplicably tied to their own i-languages.

We immediately recognize people's styles as belonging just to them, expressing their i-language, not tied to a broader visual e-language. And, people tend to get in a huff when other people "encroach" upon one's visual i-langauge. This is the taboo of copying another person's style, instead of being looked at as building or accessing a broader visual e-language.

A result of this is the belief that images have no explicit system behind them at all, since no e-language is built. Again, since drawings are iconic, and everyone's i-languages tend to differ so much, there is an appeal to perceptual knowledge as why we can understand images, instead of a specific cognitive system at work in graphic creation.

Mannerism, Imitation, and Iconic Bias

Dirk (who kindly keeps linking to my posts) provides some thoughts on my last post about imitation by describing "Mannerism" (about halfway down the post).

He describes the Renaissance Mannerists (what I'll call abstractly as "cohort 2") copied the style of the first wave of Renaissance artists like Leonardo and Michaelangelo that came before them ("cohort 1"). Where cohort 1 actually studied the anatomy of real animals and people, cohort 2 simply copied cohort 1. As a result, cohort 2 is said by Dirk to have...

"the surface tics of the Rennaissance masters down pat, but his work displays none of the anatomical understanding by which they came to be able to create such accomplished illusions of form and light. Mannerism is an artistic game of Whisper, with details lost and distorted as they move further away from their point of origin."

Dirk goes on apply this same process to describe the work of Rob Liefeld and others (who I've oddly defended on this front before).

Now, this quote to me summarizes almost exactly the points I made last time that are inherent to the "Art" perspective. With Mannerism, it's not just about copying — it's that cohort 1 didn't copy at all, they drew from real life. This is what I've called Iconic Bias in past posts (parts one, two, three): The belief that the graphic modality of expression should resemble real life ("iconicity" in the semiotic sense). The "purity" of that first cohort is drawn from iconicity, and the lack of it in cohort 2 leads to their derision.

My response is that this isn't the way the human mind is primed (the "Language" perspective): the mind is primed for imitation, and any drawing "style" is a reflection of mental patterns that have become habituated within a drawer's long-term memory. Those patterns become set in this case through one of two ways: 1) copying other people's patterns, or 2) copying perception and siphoning that iconicity through one's mental structures.

The "Art" perspective says that only choice two should be acceptable, with minimal influence from choice one. Recall for instance, McCloud's Six Steps of learning from Understanding Comics: His first level is imitation, but then all subsequent steps require one to cast aside all other influences.

But, as I've pointed out in the past, rejecting the influence of any cohort before your own works against the establishment of conventional signs — which are what language is made up entirely of. The only reason there is a "graphic dialect" of a superhero style at all is because of imitation. Manga thrive on a style that was founded on imitation (Tezuka being largely considered cohort 1, but Walt Disney and others being cohort -1 for him).

Imitation hasn't hurt manga at all. In fact, I'd argue that it has probably helped them in numerous ways: 1) A consistent cultural style allows more focus to be placed on what that style is used to express story-wise than so much focus on the surface depictions. 2) A consistent style across numerous authors is more readily accessible to young readers, especially those who want imitate them. In America, when children want to "draw comics," they want to draw stories about stuff. But, when kids want to "draw manga" they want to draw stories in the style of manga because that's the visual vocabulary that they are now exposed to.

This is just like language: "Exposure + practice = fluency." With language, successive cohorts are always the manner by which it is transmitted. A great example of this is Nicaraguan Sign Language, where several deaf children who had created their own gesture systems combined their contributions to make cohort 1. Successive cohorts took what they did only to refine and alter it into further grammatical patterns. With the anti-imitation influence of Art, this process of conventionalization is largely lost (outside symbolic signs like word balloons and speed lines at least).

The Art pespective just wants to substitute the cognitive man-made exposure for that of real life, and with that, jettisoning an idea of fluency (proficiency in a system) for skill (accuracy at depicting real life): "Perception + practice = skill at representing perception."

While I won't go into it at length, I find it intriguing that in Dirk's same post, there is a damning attitude for Greg Land, who takes iconicity to the extreme by drawing wholly from photo reference — only that he picks and chooses parts of photos to combine thereby messing up the anatomy. So, here it seems to be the case of messing up iconicity through the most iconic method possible!


Final note, so my intentions aren't misunderstood: I should point out that this is not a post of advocacy; I'm not saying people should or shouldn't copy other people. I'm just trying to analyze the issues involved, and in some case, defend all strategies as being cognitively acceptable.

----
Update 2/26: Dirk has a short reply to this post (halfway down). I don't have much in response to it except that it still maintains the "Iconic bias" underlying the last couple posts . Beyond that, he makes some interesting points.

Time essay analyzed

In Derik's continuing exploration of panel transitions (Part 1, Part 2, Part 3), he does an interesting job of dissecting my latest essay "Time Frames... Or Not." To keep things localized, I'll make my responses there, but he seems to have done a fairly thourough job of it. Worth perusing.

Also, Blogger has helpfully decided to add tags in finally, so I'll be trying to work those into all my past posts in due time.

Problems with Transitions

Over at Derik's blog he's been examining McCloud's panel transitions based on influence from film theory (Part 1, Part 2, Part 3, Part 4, Part 5 ...more to come).

While Derik only does it a little bit, the application of film theory to panel transitions isn't altogether new. John Barber essentially grafted McCloud's and my own (old model of) transitions onto Eisenstein's thesis/antithesis/synthesis model in his masters thesis. This was then argued against by Ben Woo in his thesis, dismissing it more because modern film theory does than any explicit argument against Barber's thesis. I'm not up on my modern film theory that much, but I believe Eisenstein is fairly passé at this point anyhow.

A few months ago I started noticing how similar Eisenstein's montage was to the cognitive linguistics notion of "Blending." Blending takes two concepts and extracts parts from them to create a new entailment. A classic example is "The surgeon was a butcher" — both surgeons and butchers are skilled at cutting flesh/meat, yet when combined together they illicit a meaning that the surgeon was sloppy. This is just like the 1+1=3 idea from montage.

And it certainly does appear across panels. I had a whole section on blending in my paper A Force of Change. Though, I think that the structures governing sequential understanding (i.e. syntax and semantics) are different from this.

Really, Eisenstein's montage and McCloud's closure are kind of like the film/comics equivalent of ether; a magical "mental" substance that doesn't really exist that glosses over any real substance the mind might actualy be contributing. They're like pop-science: a simple easy explanation for a very complex phenomenon. Just like Freud and Jung are still thought of by laypeople as being what psychology is about, their theories are far left behind to modern thinking. In fact, I'd venture to say they're more used by humanities/social sciences these days than psychology or cognitive science.

Of course, I've been railing on the panel transition approach for quite a while now, over the course of several alternative models. And, it's not just the idea of transitions that has problems: it's any approach that only takes into account panels that are immediately adjacent to each other. Any linear approach to the idea of creating meaning in sequential images will ultimately fail.

As I mentioned on one of Derik's posts, the major shift comes in what one is looking at. Instead of looking at panels' immediate surroundings and basing the system around those juxtapositions, we can instead acknowledge that whole sequences mean things (events/actions/situations/ideas). From there, it becomes a matter of identifying what functions different panels play in creating that overall meaning. Just because we read and write panels linearly doesn't mean that's how we understand them.

Nevertheless, it's interesting to watch Derik go through steps in his thinking in relation to what I did. He named it "rethinking transitions" so it'll be fun to see what his rethinking leads to.

Updated 12/1 with additional links to further entries

Comics software sucks

I apologize in advance for this rant...

One of the pre-installed applications on my new macbook is the "comic" making program "Comic Life." I'd played with the demo before, but figured I'd jump back in with this one to see what was changed. What didn't change was my opinion of it: not good. (It's basically a goofy photo album making program)

This has been my opinion of many of the other programs as well, ranging from Comic Book Creator to Strip Creator to Comic Creator to The Balloonist, as well as (less so) Manga Studio and Comic Works.

While some of these are very well designed programs, such as Comic Life, all of them are fundamentally deficient in the way that they are built. That is, I have the feeling that they were designed by people who have little understanding about the theory behind comics. It's one thing to have advice or commentary from people who make comics, its another to actually understand the structure of the medium and how to best utilize software to manipulate it. Especially if you want it to be used as a professional grade system, it is embarrassing not to take this into account.

(While I wouldn't be surprised if some companies consulted comic artists, I'd be very curious, for example, if Scott McCloud has been consulted on any software like this, as he is the most well known theorist out there).

For instance, a few things I noticed after using Comic Life for about half a minute...

Isn't it a bit curious that in the selection of templates for pages there aren't even standard grids, yet they do include layouts that I've experimentally seen to be problematic for readers? Why is it that when you drag in a panel, it simply appears on top of the others, and not bound within some sort of layout schema? Why do you have various templates for balloons and bubbles, instead of a generalized Carrier field that takes different representations, tails, etc. (and my god how annoying the sound effects are...)

... just to name a few. I could probably go on for pages.

Most of these programs fall into similar patterns, structuring the software as a design program... which is fine if you want to do a modified drawing program, but not if you want to be a visual writer. Even the seemingly well thought-out and evolving Comic Studio is doing things far different than I would think most useful or efficient.

As I might have mentioned before, I've had designs for a "comics" software program for about 5 years now, but have no coding skills (and little time) to work on it. So, if there's any talented and enterprising programmers out there (or companies that don't want your current product to suck so much) that would like to give it a go, feel free to drop me an email.