Whatnots

Blogging has been slow lately for me what with the oncoming start of school next week. I've mainly been devoting my time to the set-up of my latest experiment, which, after a few behavioral studies, will finally start looking at people's brainwaves reading comic strips.

To give an idea of the scope of experiment preparation: I have to create 200 novel Peanuts strips using panels from existing strips, and then make an additional 600 that are not of the "normal" variety. So, 800 new strips for one study, which will all be rated and tested in three pre-experiment studies to make sure they will be acceptable to use. Oy!

I do, however, plan to post a few reviews in the coming weeks. First, I will finally post my review of Groensteen's System of Comics, though I'm debating how best to parse it up (one LONG post? several shorter ones...?), and I may actually get around to finishing my post of Abel and Madden's latest Drawing Words & Writing Pictures.

Stay tuned.

Note: Especially when things are slow around here, I'm always open to requested topics if people want to pick my brain. Feel free to send me conversation pieces!

Equivalences for "Language"

In claiming that the graphic form (especially in sequence) is structured as a language, it might help to parse out how to make the analogy reasonably. It's not as if no one has ever noted the similarities between the forms — in fact, it's fairly common. However, mistakes are often made in my opinion.

First, I assume that there is "Equivalence" between the different modalities, which can be summarized as "the expectation that the human mind/brain treats all modalities in an equal way, given modality specific constraints."

By this account, it essentially means that we would expect all modalities to feature the same sort of storage in memory for patterned signs of differing sizes and functions (from sound patterns of phonemes to sentence patterns of idioms) and feature ways to combine all those elements at different levels of structure (phonology through discourse). We would also expect development to be similar, with a critical learning period and drop offs after that.

However, this is not the take that most comparisons of the verbals and visual forms take. Rather, they often try to make direct superficial analogies between specific types of structures. For example, "such and such" is the equivalent of a "word" or "sentence." This is often why many want to claim that single images have "grammar" — because a single image has lots of information in it, like a "sentence" and unlike a "word" — even though composition within single images behaves nothing like a grammar. (...nor should we expect it to given the differences between sound and light!)

A similar endeavor has tried to find "minimal units" of the structure of the forms, following the school of Structuralism (most popular in American linguistics from around 1920-1960ish). However, again, just knowing minimal units doesn't tell you about the broader structure, and units larger than minimal units might also be useful and insightful. It also gives no beneficial comparison other than that "minimal units" exist in both domains.

All of this is an argument for looking beyond the superficial understandings of "language" and to look for comparisons in deeper, more fundamental aspects of structuring.

Diversity in Visuals

At the VaIL conference a few weeks ago, one of the frequent conversations revolved around the issue of creating a universal graphic system.

The belief that visuals are universal is not new, and largely stems from the fact that most drawings look like what they represent ("iconicity"). This is also a motivating factor behind so-called "universal writing systems" like Blissymbols, Icon Language.

However, when looking at drawings as reflecting patterns in the mind, then there is actually quite a lot of diversity. Patterned styles like those of Japanese manga compared to stereotypical American superhero comics reflect culturally diverse conventions of different populations. Even more relative are the sand drawings of native Australian communities, and the constraints they place on recognizing other graphic depictions. Several other stories exist of people not fully understanding iconic drawings as well, such as the story Mort Walker tells about natives (I can't recall where) who thought that a person's legs were cut off, despite just being "out of frame."

Given this, perhaps it's time that we get over the idea of a universal communication system, and we come to accept that populations of humans will always develop idiosyncratic and in-group tendencies.

While globalization especially has raised a desire for inter-cultural communication, diversity in communication systems may have had evolutionary advantages. Having an identity tied to your language separate from others' means you can identify your group members, and even more helpful that you can keep secrets from other groups. If it wasn't beneficial, we would all be speaking the same languages (or so the argument could be made... diversity could just be a useful "spandrel" for the way other cognitive functions happened to turn out).

Perhaps instead of attempting to create universal systems, we should instead acknowledge the diversity that comes along with the way our brains are wired for social interactions. By accepting it, we can then strive to work with the constraints that our cognition and diversity brings or allows. That way, it will cease the fighting against the tide of inevitable diversity with rose colored glasses of universality, and instead invites appreciation for relative systems and cooperation to meet common goals of communication.

Vocab gaps

This interesting and quite fun essay/post reviews the book Reading Comics and ponders the definition of "comics" and some other terminological issues. Its biggest query is 'what is the word for the act of making a comic... the producing of sequential images bit?' (paraphrased).

I agree there's somewhat of a gap in vocabulary, but think that its partially symptomatic of a larger issue: not recognizing that drawings (especially in sequence) constitute a system of communication. That is, we lack the recognition that drawn works use a set of mental patterns the same way that, say, English does.

We in America speak in English.
We in America draw in ___?___.

Note that we also have many words describing the avenue or subroutine for picture-making: painting, sketching, penciling, inking, coloring, etc. But, no word for one actually does with those media besides the uninformative "drawing," which inadequately covers the dynamic process of creating sequences of images.

In a previous thesis of mine (¡Eye [heart] græfIk Semiosis!), I argued that vocab in a language for graphic creation partially relies on how that culture's graphic systems are structured. In Western representation, we have a dominantly sound-based "writing system" with a preference for highly "realistic" drawings — and our words for these things are very different: "Writing" versus "Drawing." Contrast this with Japan, who use a wide range of meaning/sound values in their "writing", while using much less realistic representations — and they use a single word for both concepts (kaku: かく), while using two separate Chinese characters to distinguish uses (writing: 書く, drawing: 描く).

So, in our case of English, what we're left with is a gap in vocabulary, for those representations that run a middle ground between our prototypical conceptions. Anyone who reads this blog should know my answer: The missing graphic system is what I call "visual language" (when done with images in sequence).

We in America speak in English.
We in America draw/write in American Visual Language.

If this is language, then we might as well just adopt language related vocabulary. I have no problem with the idea that I write in pictures (as well as words), and I doubt many others do either.

And, as I've argued before, recognizing that such a system exists (visual language) is the first step towards defining (or rather, un-defining) "comics." By taking away formal definitions that rely on the "comics=visual language" equation, "comics" can be defined as what we've always known it to be: socio-cultural objects (books, strips) in which a visual language is written, often pertaining to particular genres, and a culture surrounding them.

Finally, many have complained that debating vocabulary is a waste of time. But, something like this has realistic applications. Take for instance pushing the notion of visual language through the idea that sequential images are "written." Where would you learn such a thing?

You learn to write in a writing/language class, and such would be the case with visual language too. You learn to write in pictures in a writing class, not an art class — bringing it directly into the fold for education and development in a very practical and communication-driven way.

Comics, Webdesign, Closure, Cartooning

A friend of mine sent along a link to this lecture by Andy Clarke about how comics can inform webdesign. Most of the talk is a regurgitation of McCloud's theories, but he has some interesting parallels and ideas. (There's an mp3 of the talk that you can flip through the slides with).

Also, while I don't necessarily agree with the ideas, Gary Sullivan ruminates a bit on Closure.

Finally...I was sent a link to the Australian Cartooning School which has a decently formalist bent to analyzing and teaching cartooning. Definitely worth poking around the site.

Alright, heading back from San Diego soon. Perhaps I'll be able to muster up some convention/conference reports...

Talk talk talk...

I've been remiss in my reminders this year, but I have a few talks coming up if anyone is around San Diego.

The first is another appearance at the Visual and Iconic Languages Conference on July 21-22, which I believe is closed to the public (more's the pity), though my talk will be on a general overview of visual language theory. Hopefully, as with last year, they'll put the talk online.

A few days later I'll be over at Comic-Con, where my talk is on Friday at 12:30 in room 30AB. I'll be presenting a talk about manga and Japanese Visual Language. Here's the description of the panel:

12:30-2:00 COMICS ARTS CONFERENCE SESSION #7: VISUAL LANGUAGE - Neil Cohn (Tufts University) explores the visual language underlying the "manga style," how it works and how it differs from the visual languages in comics developed in other cultures. Robert O'Nale, Jr. (Henderson State University) uses David Mack’s Kabuki to illustrate how gestalt can be an important avenue for analyzing design and meaning in comics. Alec Hosterman (Indiana University South Bend) demonstrates the dominance of hyperreality in comics art and explains how it can be utilized for further study of the art form. Room 30AB

Both of the other presentations look promising, so it should be a fun panel. Come on out, enjoy the talks, and say hi!

Fun with Text

While preparing the Peanut strips for my next project, I came across a fantastic integration of text subtly hidden in this panel:



If you look carefully, at the center of the starry smack mark where the ball hit the bat, there is text reading ".315", which I assume is a reference to Pig-Pen's batting average (pretty good). This is particularly interesting to me since it's a descriptive use of text as opposed to a sound effect.

xkcd has a similar usage where instead of sound effects, the text reads the name of the action being done. However, the result in xkcd is that it describes the actions straight-out.



These two types of usage give complementary aspects of the way languages structure actions and manner of motion. Compare:

a. The ball flew into the glove.
b. The ball spiraled into the glove.

In sentence (a) you are given the action that the ball does, but little about the characteristics of that action. Sentence (b) gives you the manner of the motion, from which you derive the action implicitly.

Sound effects often give you information about manner of motion. For instance, a golf ball falling into a hole goes "Klunk" or "zoom" to describe a speeding car. These elaborate on the action itself. The use of text in xkcd eschews this to just focus on the action, without any manner of motion. What is intriguing about the Peanuts example is that ".315" is neither manner nor action — it is purely descriptive in an additive sense.

Playing with this one step further, we can create some panel pairs that replace the action for the sound effect. One characteristic of these types of "action text" is that they can stand in for the actions themselves (discussed a bit in Interfaces and Interactions). Note that both replacing for an action or manner of motion works fine:




However, substituting ".315" is a little weird — even with the expectation of the event — since it doesn't stand in for the action itself. It only gives you additional information about the action:



Looking through all these Peanuts strips, Schulz was more of a formalist than he's thought of I think.

Manga "decompression"

In Understanding Comics, McCloud made the claim that manga supposedly uses more circuitous storytelling because of the formats of their books.

In my paper on Japanese VL, I dismiss this on the grounds that nothing about longer formats gives people the drive to make slower paced narrative. Just because you have ample space doesn't mean you're going to use it to let the story linger more. You could use that space to fill in even more "compressed" storytelling.

Thinking more about this, webcomics are another good example against this theory: from my knowledge, we haven't seen a vast decompression of storytelling on the web due to the completely unrestricted "infinite" space allowing authors to freely use (though feel free to prove me wrong!). On the one hand, you could say that they aren't effectively using the space that they have at their disposal. However, the other side could say that they're using it to achieve just what they want: they have no restrictions, so what they're producing is entirely their preference.

Personally, I think that there are numerous explanations for what might be going on in manga storytelling. Here's a few, some of which were in my paper...

1) It's just an inherent part of the difference between Euro-American VLs and Japanese VLs. We don't expect spoken languages to be the same, why should visual languages? Could "decompression" simply be a result of the development of how JVL evolved?

2) They're using the VL as a language: Manga use less text than American and European books. With more reliance on visual modality over the written requires it to take on more expressive weight. The result is more complex structure in the visual sequences. This is comparable to studies asking people to only gesture with no speaking. The result is something that looks closer to patterns like in sign languages (though still not SL).

3) The cross-cultural differences focusing on environment over action requires more space devoted to "setting a scene." Research seems to suggest that Asian minds are more interested in the broader environment than the specifics and individuating different elements of the environment take up more panel space than simply presenting it as a whole, backgrounded to the actions.

Notice that in all of these cases, formatting is entirely secondary. Indeed, it's somewhat interesting to think that formatting is one of McCloud's explanations, because much of his work is about transcending formatting. Here, the explanations focus on cognitive reasoning — meaning we should see the effects no matter what the format.

Garfield experimentalism

Apparently we're upon the 30th anniversary of Jim Davis' Garfield strip. As a ten year old I was pretty obsessed with the Garfield books, and can probably mark meeting Jim Davis at the ABA as a highlight of my fourth grade life.

Perhaps unsurprisingly, I've gotten about seven emails from people linking to the Garfield Minus Garfield strips, which I first saw a few years ago even. I was always partial to the Is Garfield Dead? premise, though Nothing Garfield strips are interesting too (though Barfield does give me a good chuckle).

More theory related, the Garfield Generator is a great example of a few points of my research. It shows that there is an overarching coherent structure built into the whole strip (at least sometimes in this case), even when the immediate linear relationships don't make much sense. This is somewhat similar to the famous Chomsky sentence "Colorless green ideas sleep furiously", and I'm actually basing my next big experiment out of 6 panel long Peanuts panels of this same nature.

In some cases with the generator though, you can easily tell that the position of the panel is somewhere it doesn't belong. The thematic role of the panel belies it's canonical positioning.

Anyhow, Happy Birthday Garfield, and thanks for the early influences on my comics obsessions!

Comics and the Brain... almost

Nagai, Masayoshi, Nobutaka Endo, and Kumada Takatsune. 2007. "Measuring Brain Activities Related to Understanding Using near-Infrared Spectroscopy (Nirs)." In Human Interface and the Management of Information. Methods, Techniques and Tools in Information Design, 884-93. Heidelberg: Springer Berlin

Looks like I was beat to the punch... I've found a study from last year that analyzes the activity in the brain while reading comics. However, it doesn't say much.

The authors use near-infrared spectroscopy to measure blood flow in the brain while reading comics. This technique uses infrared light to measure where blood flows in the brain, which can thus indicate the brain regions involved in various behaviors. They find that "the left prefrontal lobe region is activated when people actively try to understand the comic stories and to memorize their contents for reporting in the future."

However, there are extensive problems with this study. First, the number of stimuli they use is extremely small (only 6 strips) as is their population (13 people... which does not add up to counterbalancing). Comparatively, the study I'm planning will use 180 stimuli per trial (720 strips total) and use somewhere from 24 to 36 people.

Additionally, the increase in blood flow that they observe only occurs in "reported" conditions — where subjects are actively making a judgement about the stimuli, as opposed to scenarios when they are not. This seems more to reflect the well-reported cognitive activation for making judgements than anything about the structure of the comics themselves.

So... this really doesn't tell us much about comics and the brain, but its nice to see other people are at least taking stabs at it as well.

Collaborative drawing

Last weekend on public television I saw a fantastic biography about Pete Seeger, the influential folk singer and activist. Throughout, Seeger stressed his desire to sing with people, not to people — motivating music as a collaborative endeavor. This sentiment is echoed in the accessible book, This is Your Brain on Music, which points out that music as "performance" by people on a stage to other people seems to be a fairly new thing. Traditionally, music was a group activity that was not reserved for those of express "skill" and training.

Drawing is much the same way. We often make a huge break between those with or without "talent" — resigning people to the misperception that they "can't draw", when really our biological endowment ensures that we all can draw. Really what is at issue is a level of fluency, and most people just don't develop with the proper exposure or practice.

Language, like this sense of music, is entirely collaborative. And, it is learned collaboratively, unlike most learning of drawing. In some cases, drawing might be instructed, often very well, though this is far from simply being interactive in the sense that you learn just by participatory immersion.

On a productive sense, drawing also is highly non-collaborative in our modern life. Belonging to a print-culture, most drawers and readers are separated by huge distances of space and time. This isn't always the case though. Sand narratives by native communities in Australia are highly interactive, drawn in real-time communication.

Humans are an intensely social animal, and my gut tells me that nearly all of our expressive capacities developed and thrive in such collaborative interactions. The question is: how in our modern ecology can we facilitate such usage for visual language? Will we have to rely on technological breakthroughs (ex. digital whiteboards), or can it grow organically without the crutch of engineering?


For those interested in more about this, my article from a few years ago "Interactive Comics" probed a lot of these ideas.

Comics as a Binary Language

Laraudogoitia, Jon Pérez. 2008. The Comic as a Binary Language: An Hypothesis on Comic Structure. Journal of Quantitative Linguistics 15 (2):111-135.

This study examines the structure of comics by converting the contents of panels into binary code. Coding a broad number of Eurpoean comics, a panel holding the protagonist of a story ("lead character") is given a "+" while a panel without is given a "-". The author then uses a series of computations to examine the regularity of sequences where the protagonist does or does not appear, or if there is constancy to the amount that they appear througout a book.

The results show that there is a quasi-regularity to sequences that feature the protagonist or don't feature the protagonist. That is, there are "runs" of sequences with protagonists, then runs without.

While interesting for coming up with a positive result — and very creative for applying computational methods to comics (somehting I don't think has otherwise been done), I find numerous problems with this paper.

First, why should we assume that Protagonist vs. Non-protagonist is a meaningful binary juxtaposition? In some ways it reflects of my distinction between Active and Inactive (or Passive) entities in a panel (originally based on Natsume's distinction of "positive" vs. "negative" entities). However, my breakdown is superficially "things that move across panels" to "things that don't." Protagonists could fall into either one of those categories given the appropriate sequence.

But... what if there is more than one protagonist? What if a scene shift happens where a new character becomes the lead character — this would just be coded as a consistent "-"?

Mostly though, I am unsure of what is interesting about these results. The visual language in comics features consistent "runs" of protagonist or non-protagonist panels: so what?

The analysis throughout focuses only on linear sequences based largely on Markovian chains, but I think my work has strived to show that sequences of images cannot simply be considered linear sequences. They have hierarchic structures guiding them — which such a binary analysis of the surface elements would be unable to show.

This study is an interesting first attempt at using computational methods to analyze visual language structure — and I love that the research has now begun permeating such extants. Hopefully further studies will bring more interesting results.