Talk talk talk...

I've been remiss in my reminders this year, but I have a few talks coming up if anyone is around San Diego.

The first is another appearance at the Visual and Iconic Languages Conference on July 21-22, which I believe is closed to the public (more's the pity), though my talk will be on a general overview of visual language theory. Hopefully, as with last year, they'll put the talk online.

A few days later I'll be over at Comic-Con, where my talk is on Friday at 12:30 in room 30AB. I'll be presenting a talk about manga and Japanese Visual Language. Here's the description of the panel:

12:30-2:00 COMICS ARTS CONFERENCE SESSION #7: VISUAL LANGUAGE - Neil Cohn (Tufts University) explores the visual language underlying the "manga style," how it works and how it differs from the visual languages in comics developed in other cultures. Robert O'Nale, Jr. (Henderson State University) uses David Mack’s Kabuki to illustrate how gestalt can be an important avenue for analyzing design and meaning in comics. Alec Hosterman (Indiana University South Bend) demonstrates the dominance of hyperreality in comics art and explains how it can be utilized for further study of the art form. Room 30AB

Both of the other presentations look promising, so it should be a fun panel. Come on out, enjoy the talks, and say hi!

Fun with Text

While preparing the Peanut strips for my next project, I came across a fantastic integration of text subtly hidden in this panel:



If you look carefully, at the center of the starry smack mark where the ball hit the bat, there is text reading ".315", which I assume is a reference to Pig-Pen's batting average (pretty good). This is particularly interesting to me since it's a descriptive use of text as opposed to a sound effect.

xkcd has a similar usage where instead of sound effects, the text reads the name of the action being done. However, the result in xkcd is that it describes the actions straight-out.



These two types of usage give complementary aspects of the way languages structure actions and manner of motion. Compare:

a. The ball flew into the glove.
b. The ball spiraled into the glove.

In sentence (a) you are given the action that the ball does, but little about the characteristics of that action. Sentence (b) gives you the manner of the motion, from which you derive the action implicitly.

Sound effects often give you information about manner of motion. For instance, a golf ball falling into a hole goes "Klunk" or "zoom" to describe a speeding car. These elaborate on the action itself. The use of text in xkcd eschews this to just focus on the action, without any manner of motion. What is intriguing about the Peanuts example is that ".315" is neither manner nor action — it is purely descriptive in an additive sense.

Playing with this one step further, we can create some panel pairs that replace the action for the sound effect. One characteristic of these types of "action text" is that they can stand in for the actions themselves (discussed a bit in Interfaces and Interactions). Note that both replacing for an action or manner of motion works fine:




However, substituting ".315" is a little weird — even with the expectation of the event — since it doesn't stand in for the action itself. It only gives you additional information about the action:



Looking through all these Peanuts strips, Schulz was more of a formalist than he's thought of I think.

Manga "decompression"

In Understanding Comics, McCloud made the claim that manga supposedly uses more circuitous because of the formats of their books.

In my paper on Japanese VL, I dismiss this on the grounds that nothing about longer formats gives people the drive to make slower paced narrative. Just because you have ample space doesn't mean you're going to use it to let the story linger more. You could use that space to fill in even more "compressed" storytelling.

Thinking more about this, webcomics are another good example against this theory: from my knowledge, we haven't seen a vast decompression of storytelling on the web due to the completely unrestricted "infinite" space allowing authors to freely use (though feel free to prove me wrong!). On the one hand, you could say that they aren't effectively using the space that they have at their disposal. However, the other side could say that they're using it to achieve just what they want: they have no restrictions, so what they're producing is entirely their preference.

Personally, I think that there are numerous explanations for what might be going on in manga storytelling. Here's a few, some of which were in my paper...

1) It's just an inherent part of the difference between Euro-American VLs and Japanese VLs. We don't expect spoken languages to be the same, why should visual languages? Could "decompression" simply be a result of the development of how JVL evolved?

2) They're using the VL as a language: Manga use less text than American and European books. With more reliance on visual modality over the written requires it to take on more expressive weight. The result is more complex structure in the visual sequences. This is comparable to studies asking people to only gesture with no speaking. The result is something that looks closer to patterns like in sign languages (though still not SL).

3) The cross-cultural differences focusing on environment over action requires more space devoted to "setting a scene." Research seems to suggest that Asian minds are more interested in the broader environment than the specifics and individuating different elements of the environment take up more panel space than simply presenting it as a whole, backgrounded to the actions.

Notice that in all of these cases, formatting is entirely secondary. Indeed, it's somewhat interesting to think that formatting is one of McCloud's explanations, because much of his work is about transcending formatting. Here, the explanations focus on cognitive reasoning — meaning we should see the effects no matter what the format.

Garfield experimentalism

Apparently we're upon the 30th anniversary of Jim Davis' Garfield strip. As a ten year old I was pretty obsessed with the Garfield books, and can probably mark meeting Jim Davis at the ABA as a highlight of my fourth grade life.

Perhaps unsurprisingly, I've gotten about seven emails from people linking to the Garfield Minus Garfield strips, which I first saw a few years ago even. I was always partial to the Is Garfield Dead? premise, though Nothing Garfield strips are interesting too (though Barfield does give me a good chuckle).

More theory related, the Garfield Generator is a great example of a few points of my research. It shows that there is an overarching coherent structure built into the whole strip (at least sometimes in this case), even when the immediate linear relationships don't make much sense. This is somewhat similar to the famous Chomsky sentence "Colorless green ideas sleep furiously", and I'm actually basing my next big experiment out of 6 panel long Peanuts panels of this same nature.

In some cases with the generator though, you can easily tell that the position of the panel is somewhere it doesn't belong. The thematic role of the panel belies it's canonical positioning.

Anyhow, Happy Birthday Garfield, and thanks for the early influences on my comics obsessions!

Comics and the Brain... almost

Nagai, Masayoshi, Nobutaka Endo, and Kumada Takatsune. 2007. "Measuring Brain Activities Related to Understanding Using near-Infrared Spectroscopy (Nirs)." In Human Interface and the Management of Information. Methods, Techniques and Tools in Information Design, 884-93. Heidelberg: Springer Berlin

Looks like I was beat to the punch... I've found a study from last year that analyzes the activity in the brain while reading comics. However, it doesn't say much.

The authors use near-infrared spectroscopy to measure blood flow in the brain while reading comics. This technique uses infrared light to measure where blood flows in the brain, which can thus indicate the brain regions involved in various behaviors. They find that "the left prefrontal lobe region is activated when people actively try to understand the comic stories and to memorize their contents for reporting in the future."

However, there are extensive problems with this study. First, the number of stimuli they use is extremely small (only 6 strips) as is their population (13 people... which does not add up to counterbalancing). Comparatively, the study I'm planning will use 180 stimuli per trial (720 strips total) and use somewhere from 24 to 36 people.

Additionally, the increase in blood flow that they observe only occurs in "reported" conditions — where subjects are actively making a judgement about the stimuli, as opposed to scenarios when they are not. This seems more to reflect the well-reported cognitive activation for making judgements than anything about the structure of the comics themselves.

So... this really doesn't tell us much about comics and the brain, but its nice to see other people are at least taking stabs at it as well.

Collaborative drawing

Last weekend on public television I saw a fantastic biography about Pete Seeger, the influential folk singer and activist. Throughout, Seeger stressed his desire to sing with people, not to people — motivating music as a collaborative endeavor. This sentiment is echoed in the accessible book, This is Your Brain on Music, which points out that music as "performance" by people on a stage to other people seems to be a fairly new thing. Traditionally, music was a group activity that was not reserved for those of express "skill" and training.

Drawing is much the same way. We often make a huge break between those with or without "talent" — resigning people to the misperception that they "can't draw", when really our biological endowment ensures that we all can draw. Really what is at issue is a level of fluency, and most people just don't develop with the proper exposure or practice.

Language, like this sense of music, is entirely collaborative. And, it is learned collaboratively, unlike most learning of drawing. In some cases, drawing might be instructed, often very well, though this is far from simply being interactive in the sense that you learn just by participatory immersion.

On a productive sense, drawing also is highly non-collaborative in our modern life. Belonging to a print-culture, most drawers and readers are separated by huge distances of space and time. This isn't always the case though. Sand narratives by native communities in Australia are highly interactive, drawn in real-time communication.

Humans are an intensely social animal, and my gut tells me that nearly all of our expressive capacities developed and thrive in such collaborative interactions. The question is: how in our modern ecology can we facilitate such usage for visual language? Will we have to rely on technological breakthroughs (ex. digital whiteboards), or can it grow organically without the crutch of engineering?


For those interested in more about this, my article from a few years ago "Interactive Comics" probed a lot of these ideas.

Comics as a Binary Language

Laraudogoitia, Jon Pérez. 2008. The Comic as a Binary Language: An Hypothesis on Comic Structure. Journal of Quantitative Linguistics 15 (2):111-135.

This study examines the structure of comics by converting the contents of panels into binary code. Coding a broad number of Eurpoean comics, a panel holding the protagonist of a story ("lead character") is given a "+" while a panel without is given a "-". The author then uses a series of computations to examine the regularity of sequences where the protagonist does or does not appear, or if there is constancy to the amount that they appear througout a book.

The results show that there is a quasi-regularity to sequences that feature the protagonist or don't feature the protagonist. That is, there are "runs" of sequences with protagonists, then runs without.

While interesting for coming up with a positive result — and very creative for applying computational methods to comics (somehting I don't think has otherwise been done), I find numerous problems with this paper.

First, why should we assume that Protagonist vs. Non-protagonist is a meaningful binary juxtaposition? In some ways it reflects of my distinction between Active and Inactive (or Passive) entities in a panel (originally based on Natsume's distinction of "positive" vs. "negative" entities). However, my breakdown is superficially "things that move across panels" to "things that don't." Protagonists could fall into either one of those categories given the appropriate sequence.

But... what if there is more than one protagonist? What if a scene shift happens where a new character becomes the lead character — this would just be coded as a consistent "-"?

Mostly though, I am unsure of what is interesting about these results. The visual language in comics features consistent "runs" of protagonist or non-protagonist panels: so what?

The analysis throughout focuses only on linear sequences based largely on Markovian chains, but I think my work has strived to show that sequences of images cannot simply be considered linear sequences. They have hierarchic structures guiding them — which such a binary analysis of the surface elements would be unable to show.

This study is an interesting first attempt at using computational methods to analyze visual language structure — and I love that the research has now begun permeating such extants. Hopefully further studies will bring more interesting results.

My "Homopholganger"

Huzzah! Today is the sixth year this website has been online! If I remember correctly, I posted everything online while the Lakers were in the playoffs about to go to the championships... and lo and behold the past is repeating! (yes, I'm a Laker fan... which will certainly be interesting living in Boston as they move on to play the Celtics in the Finals)

So, here's a semi-research-related story to commemorate the occasion. The Tufts Psych department (to which I'm a grad student) is hosting a conference this weekend, and one of the featured speakers is a psychologist named...Neal Cohen! (no relation)

Naturally, I thought it would be hilarious and awkward to meet him. The first day of the conference I had turned it into a scavenger hunt, with several faculty and other grad students all on the lookout for him. We came up with a great portmanteau word to describe someone who shares the same name as you: your "homopholganger." By the time I arrived Friday, I was getting asked over and over if I'd met him yet.

I actually did end up talking to him shortly after his own presentation, and hilarity ensued! Even cooler, I started making connections between some of his work on the hippocampus to things I'm finding in visual grammar. Naturally, I proposed a collaboration... He thought the ideas were pretty cool, so, who knows, perhaps in the next few years we'll see the fantastic byline: By Neil Cohn and Neal Cohen.

Random!... panel sequences that is

As long as we're on the topic of comics that people clip out for me, here's another one that my advisor passed along. For some reason, he's rather partial to Zippy the Pinhead (I think because of the philosophy jokes), and this one caught his eye. Particularly this first panel over to the side.

Zippy it seems comes from the Non-sequitur school of panel transitions (if you're into that sort of thing).

What makes this fun for me is that my next experiment is actually going to use various scrambled strips to help illustrate the differences in processing between those and normal strips (plus some other more complex strip types).

Not much is out there about this sort of research, but one study did show that people's comprehension of sequential "picture stories" (Mercer Mayer stories) correlated with their comprehension for text. Skilled readers showed a drop in recollection for scrambled compared to regular sequences. However, unskilled readers showed no comprehension differences at all.

I'm a bit dubious that fluency in visual language is comparable to general comprehension skills (they used no measure for graphic fluency), but this study at least showed some support for a domain general capacity.

Rory Root, we'll miss you

I learned to great shock and sorrow this morning that Rory Root, owner and operator of Comic Relief in Berkeley, has passed away. Rory was a phenomenal presence in the comic industry, and I remember fondly being first introduced to him by Beau Smith as having "the best comic store anywhere" when I was still working for TMP at the ComicCon as a teenager. When Beau discovered I was going to go to college at UC Berkeley, he made sure I knew Rory before going.

During my time at Berkeley, Rory was always interesting and encouraging, especially when I began my greater foray into theory. His was the first store that carried my Early Writings on Visual Language book on theory, and always made sure I did booksignings with them at ComicCon.

Rory once took me to lunch on the auspices of giving advice for future bookselling. He was always quick to introduce me to people he thought might give me good exposure, one time unexpectedly taking one of my books out of my hands to give to a blogger saying "Trust me, this will be good publicity." (He was right)

He was a fixture in this industry, and a wonderful friend and benefactor. He will be greatly missed.

Split planes

I haven't gotten a newspaper at home in years, but every now and then my father sends me clippings of comics or articles. The comic that he seems to send the most is Pearls Before Swine which has a periodic flair for formalism.

In the last set that he sent, the characters do a fair amount of walking on the borders of the panels (Here, Here, and Here):



"Awareness" of panel borders by characters within them is nothing new, but doing so it reveals that there are two levels of representations in this visual language of comics. There is a "Representational Plane" (RP) that the content exists in, and a "Framing Plane" (FP) that holds things like panel borders and balloons/bubbles/text boxes. Usually, the Framing Plane just lies "outside" the RP, but instances like these collapse the layers together. (see linked essay below for illustrations of this)

Another hint that these two layers exist comes from the fact that text carriers can become panels, as I discussed in my article on "Loopy Framing":



This commonality between their forms — that they both encapsulate information, both are not part of the image matter but can be interacted with in a "meta" way — go towards their being two aspects of a singular plane of Framing.

Note: For those more interested, I discuss this more extensively in my paper Interactions and Interfaces.

On PowerPoint

I was recently asked by the VizThink people to devote a post to the prompt: PowerPoint: A powerful tool poorly used or a poor tool overused?, so... here goes...

I do think that PowerPoint** can be a powerful tool that is misused, but its functions and autofills can also be overused. In many ways this is not the fault of the program, but the fault of the users for relying on the program to guide their presentations and thinking. (though, perhaps some blame should go to the programmers who design it to serve that purpose)

At its heart, the program is just a slide show. Its key function is to show slides one at a time after each other. Everything else is just bells and whistles. That potential is extremely powerful, but it can be misused. The choice of what to put on those slides makes all the difference. This is a reflection of the user — and their own abilities for storytelling, narrative, etc.

When slides are used as an alternative to substantial speaking they become a hinderance: Cramming too much information on a slide. Presenting information that people will be forced to digest at the same time as trying to follow your speech. Seeing the presentation as a rigid path not allowing the free form creativity of writing on a blackboard.

From my experience, slides should be used like gestures. Co-speech gesture occurs at a rate of roughly one gesture per clause, and usually elaborates upon some aspect of speech often adding a spatial dimension to it. It isn't something separate that happens to go with speech though — it expresses an integral and complementary part of the conceptualization that also goes into verbal expression.

Another analogy more pertinent to this blog (and one I discuss in this podcast), is that slides should serve the equivalent of a "panel." The slide is the image content of a panel, while your speech is like the text. The biggest difference between the two is the temporal quality of presenting them — otherwise they serve largely the same function.

Not all use of slide shows need to be clipped and truncated as the Powerpoint Gettysburg suggests. You can still have beautiful and powerful oration using slides — but it should not depend on the slides. Rather, the slides let speech be more than just sounds. It has to be a multimodal expression, where both slides and speech work in concert with each other to achieve something more. The ability to do this, I would suspect, is cognitively the same whether it's done in print or on a screen.

PowerPoint is not a substitute for lack of narrative skills, and its problems can largely be fingered for forgetting or believing that (whether as a user or a programmer). Excel can't make you a powerful statistician. Word won't make you a good writer. Why should we expect that PowerPoint is to blame for poor presentations?


**My personal preference is actually Keynote. I'm going to treat this as just a discussion of slide show programs in general. If we're targeting PowerPoint specifically, then you can amp up my dislike of the autofills, etc.