Garfield experimentalism

Apparently we're upon the 30th anniversary of Jim Davis' Garfield strip. As a ten year old I was pretty obsessed with the Garfield books, and can probably mark meeting Jim Davis at the ABA as a highlight of my fourth grade life.

Perhaps unsurprisingly, I've gotten about seven emails from people linking to the Garfield Minus Garfield strips, which I first saw a few years ago even. I was always partial to the Is Garfield Dead? premise, though Nothing Garfield strips are interesting too (though Barfield does give me a good chuckle).

More theory related, the Garfield Generator is a great example of a few points of my research. It shows that there is an overarching coherent structure built into the whole strip (at least sometimes in this case), even when the immediate linear relationships don't make much sense. This is somewhat similar to the famous Chomsky sentence "Colorless green ideas sleep furiously", and I'm actually basing my next big experiment out of 6 panel long Peanuts panels of this same nature.

In some cases with the generator though, you can easily tell that the position of the panel is somewhere it doesn't belong. The thematic role of the panel belies it's canonical positioning.

Anyhow, Happy Birthday Garfield, and thanks for the early influences on my comics obsessions!

Comics as a Binary Language

Laraudogoitia, Jon Pérez. 2008. The Comic as a Binary Language: An Hypothesis on Comic Structure. Journal of Quantitative Linguistics 15 (2):111-135.

This study examines the structure of comics by converting the contents of panels into binary code. Coding a broad number of Eurpoean comics, a panel holding the protagonist of a story ("lead character") is given a "+" while a panel without is given a "-". The author then uses a series of computations to examine the regularity of sequences where the protagonist does or does not appear, or if there is constancy to the amount that they appear througout a book.

The results show that there is a quasi-regularity to sequences that feature the protagonist or don't feature the protagonist. That is, there are "runs" of sequences with protagonists, then runs without.

While interesting for coming up with a positive result — and very creative for applying computational methods to comics (somehting I don't think has otherwise been done), I find numerous problems with this paper.

First, why should we assume that Protagonist vs. Non-protagonist is a meaningful binary juxtaposition? In some ways it reflects of my distinction between Active and Inactive (or Passive) entities in a panel (originally based on Natsume's distinction of "positive" vs. "negative" entities). However, my breakdown is superficially "things that move across panels" to "things that don't." Protagonists could fall into either one of those categories given the appropriate sequence.

But... what if there is more than one protagonist? What if a scene shift happens where a new character becomes the lead character — this would just be coded as a consistent "-"?

Mostly though, I am unsure of what is interesting about these results. The visual language in comics features consistent "runs" of protagonist or non-protagonist panels: so what?

The analysis throughout focuses only on linear sequences based largely on Markovian chains, but I think my work has strived to show that sequences of images cannot simply be considered linear sequences. They have hierarchic structures guiding them — which such a binary analysis of the surface elements would be unable to show.

This study is an interesting first attempt at using computational methods to analyze visual language structure — and I love that the research has now begun permeating such extants. Hopefully further studies will bring more interesting results.

Random!... panel sequences that is

As long as we're on the topic of comics that people clip out for me, here's another one that my advisor passed along. For some reason, he's rather partial to Zippy the Pinhead (I think because of the philosophy jokes), and this one caught his eye. Particularly this first panel over to the side.

Zippy it seems comes from the Non-sequitur school of panel transitions (if you're into that sort of thing).

What makes this fun for me is that my next experiment is actually going to use various scrambled strips to help illustrate the differences in processing between those and normal strips (plus some other more complex strip types).

Not much is out there about this sort of research, but one study did show that people's comprehension of sequential "picture stories" (Mercer Mayer stories) correlated with their comprehension for text. Skilled readers showed a drop in recollection for scrambled compared to regular sequences. However, unskilled readers showed no comprehension differences at all.

I'm a bit dubious that fluency in visual language is comparable to general comprehension skills (they used no measure for graphic fluency), but this study at least showed some support for a domain general capacity.

Podcast: "Grammar" in visual language

I've done another podcast with the folks at VizThink, this time debating Yuri Engelhardt and Dave Gray on what constitutes a visual language and the nature of visual language grammar.

This new format allows you to skip around to different chapters to jump straight to parts of interest. (Please note, I object to the insinuation in the chapter title that "comics" can equal "visual language"):

Hint: Use the Full Screen Button to see this video in greater detail.

I think that there is something I strived to point out throughout the discussion that I didn't articulate well enough, but to explain it I'll have to do a mini-linguistics lesson.

In the podcast, Yuri pointed out the view that language has two main parts: a set of units (lexicon) and a set of combinatorial rules (grammar). This view of two components is essentially Chomsky's view of grammar, and organizationally looks something like the diagram to the side.



In this traditional view, syntax/grammar is the component that offshoots meaning, and only syntax has properties for combining elements together. I said that I agreed with this notion, but really I don't. When I mentioned that I subscribe to a view from Chomsky's student, Ray Jackendoff (my teacher), I should perhaps have elaborated on the differences between those perceptions more, because they are extremely important and can resolve some of the conflict of the debate.

Jackendoff's view of grammar is different. This "Parallel Architecture" says that the mind has three main interfacing components: modality (auditory/manual/graphic), syntax, and conceptual structure (meaning). The "lexicon" is distributed across the interfaces between all three of these structures — it doesn't have it's own "place." And, importantly, each of these structures has that capacity for infinite combinations — not just syntax. (Note the similarities to my listing of properties of Language). This would look like this:



Much of our debate focused around whether or not single images (diagrams) have "grammar." My objection is that it does not function like "syntax" does in a verbal grammar, though I acknowledge that there might be a hierarchy or a combinatorial system there. If you subscribe to the Chomksyan view of grammar, you're forced to say that the combinatorial element "is syntax," which is exactly what Yuri is doing:



If you follow the Parallel Architecture (as I do), syntax is not the only element that creates hierarchies. They all do. So, combinations within a single image or diagram is "grammar' insofar as phonology is the "grammar" of sound. Essentially, Yuri's "visual grammar" is the combination system within the graphic structures, which is why I kept prodding about the difference between it and just the system of perception (and why most of its "constraints" are based on iconicity). This instead looks like this:



In contrast, my grammar for visual language needs a combinatorial system for individual images and for combining them together, looking like this:



To the extant that the narrative structure takes concepts and a modality and orders them coherently, it functions the same as syntax in verbal language. This is "visual language grammar" analogous to the way that syntax is verbal language grammar (nouns and verbs). But, all three structures have combinatorial properties. They don't all make reasonable analogies to saying that they are like "grammar" in the syntactic sense, but they may be combinatorial.

(This is also why you can say that "gestures are to sign language what individual images are to visual language" in the context of sequential images, but not for individual images. There is no developmental/fluency gap like this for "... visual objects are to individual images". I.e. People don't learn how to draw simple graphic signs but not be able to put them into a diagrammatic arrangement.)

Making this shift in perception buys you a lot: It makes the distinction why single images may have hierarchy (like perception/phonology), but don't have grammar (like syntax). It addresses why most of that structure is guided by iconic and indexical constraints. And, it also may give you a leg up in describing combinatorial aspects of images beyond diagrams (which occurs within panels).

Finally, it is worth noting that not just aspects of language have consistent patterned units that appear hierarchic in structure within our cognitive system. This also appears in music, event structure, vision, social structure, and a myriad of other domains (discussed well here). But, we don't have to call them "languages" because of this broad similarity.

Suggested reading:
Foundations of Language and
Language, Consciousness, Culture, both by Ray Jackendoff

Time and The Torch

On this page I found another great example of a page by Jae Lee that defies the "temporal mapping" idea that successive panels are successive moments:

I'm unaware of the full context of the page, but the Human Torch is flying around some big monster of sorts and creates the number "4" (for Fantastic Four no doubt) in his path. Doing so, his path begins by violating a constraint of page layout, entering at the bottom of the page, and then flies over his own path, which crosses a panel he's already been in.

I'm not sure I agree with the analysis given on that blog, mainly because I think appealing to McCloud's transitions and closure only hurts his otherwise fairly good discussion.

Now, I don't want to suggest here that there is not time being shown here, but I think that there are two considerations that need to be reoriented.

First, let's not talk about "time," let's talk about "events." To the human mind time is only an extrapolation of events. Thinking in terms of a clicking-clock type of absolutist Time is not on the same level with the understanding of time constructed in a person's head. From understanding events, we can tell that time passes, not so the other way around.

Second, panels do not necessarily have to equal moments. Rather, panels function as "attention units" grouping important information into meaningful chunks. These chunks don't have to be moments, but they do highlight relevant information in ways that the author intends.

This is exactly the case in this example. The interesting thing is that the flow of events runs counter to the standard reading path of panels in order to create the "4" emblem. If reading left-to-right as if these were independent moments, this would make no sense whatsoever. But, because this display uses image constancy (breaking up a single image into parts... what I'd call a Divisional panel, the understanding of which is what Gestalt psychology would call Closure), the panels only serve to divide up the conceptual space of the image to highlight the Torch at different positions within the space.

Yes, the countering of events vs. panels is a bit funky, but it's also a creative use of playing the two off each other to reveal their functions.


Note: For those more interested in these types of examples about Time, most of these ideas are written about more extensively in my paper Time Frames... Or Not. Attention Units are discussed more in A Visual Lexicon.

Some links and whatnots

Steven Seagle has a decent piece up at the First Second blog about visual storytelling. He nicely taps into a simplified version of some of the same things that I've been pushing for my theory of visual grammar. The exercise he uses to rearrange panels is very reminiscent of linguistics methods, and is also a good one that shows how a broader structure exists above and beyond the so-called 'transitions' between panels.

Along these lines, Matt Madden and Jessica Abel will soon have a "how to" book coming out about comics. I've been hearing that its somewhat theory oriented, so the book should be an interesting read. So, keep an eye out in the coming months.

Finally, keep an eye on this very site in the next week. I will finally — finally — be posting the results of my experiment about comic page layouts shooting for next Monday. This one has been a long time coming — I first ran the experiment almost 4 years ago and have been working on the paper since last spring! The project tested whether or not people read comic page layouts using the "left-to-right and down" path like text. A preview: the answer is "not really."

Finally, last month was my most trafficked month ever, so, thanks to everyone that's been reading my site lately!

Closure's assumptions

Patric continues his defining of "comics" with a discussion of "closure." I've talked before about the problems with the idea of closure, but it strikes me that there are a few underlying issues that people run into when addressing these issues:

1. They assume that time passes between panels, despite there being no evidence that each panel represents a "moment in time." With this assumption in place, it forces people to assume that some "moment" also lies between the panels, when no hidden moment may exist. I wrote my essay "Time Frames...Or Not" about various reasons why this assumption isn't true.

Even McCloud bungles this. While in one place he tries to say that "panels=moments" because "time=space", in his own transitions he includes three that have nothing to do with time at all! (Subject, Aspect, and Non-Sequitur transitions). For the adamant, what are the moments and what are the transitions in this "comic"?

2. People are just looking at the relationships of two juxtaposed panels. Most stabs at sequential meaning, like Patric's or Derik's, have just talked about two-panel pairs. But, rarely are sequences confined to two panels.

Just because we experience reading sequences of images linearly doesn't mean that is how we understand them. In most cases, we can easily acknowledge that whole sequences mean something beyond just paired panels. Looking beyond the scope of immediate panel relations quickly forces a rethinking of the accuracy of a view about closure/transitions.

Here are a few illustrative exercises that people can do to think more about these issues (and are things I did when first getting into this seriously):

1. Actually try to catalogue the "transitions" in a full comic á la McCloud's counting. Note any problems in the categories and where descriptions become more difficult.

2. Take comic pages/strips and sketch out the different relationships of every panel to each other. Which panels need connections, which don't? What do the relationships tell you?

If anyone actually does this, I'd love to hear about their results. In the meantime, if people are curious about my alternatives to closure, I recommend watching this video.

Podcast: The Functions of Panels

The last podcast I did with the VizThink folks was so fun I decided to do another. This one is about the various functional roles that panels play in the visual language used in comics. Among the topics I hit are:

• focusing information within panels
• navigating page layouts
• visual "storytelling"
• text-image relationships

It's a slightly pared down and also expanded (at the same time!) version of the talk I gave at the VizThink conference. Enjoy!

Connectedness in comics

Weber, Heinz J. 1989. Elements of Text-Based and Image-Based Connectedness in Comic Stories, and Some Analogies to Cinema and Written Text. Paper read at Text and Connectedness: Proceedings of the Conference on Connexity and Coherence, July 16-21 1984, at Urbino, Italy.

Weber attempts to create a textual model of comic communication drawing on cinema studies and text research (similar to the intents of Saraceni and others).

He describes three sections: Graphic, cinematographic, and textual, as well as the intersections between them. He also postulates several degrees of “connectedness” ranging from conformity, sequential and integrated connexity, cohesion, and coherence.

Conformity deals with arrangement of panels – conventionalized formats, while connexity can either relate to the internal relationships of elements within panels, layout issues, divisional panels, metonymic panels, the shifting of a balloon's tail to different roots, etc. Cohesion depends on causality between the succession of panels (syntax/semantics). Finally, coherence deals with pragmatic relations between “text external” elements.

Like other papers of this ilk, it provides a broad scale analysis of aspects of comics' structure, yet doesn't delineate them carefully enough to detail the componential role they might play. Instead, it uses overarching "principles" to tie them all together to create a goolash of comic structure.

Some Peanuts patterns

It's the end of the semester, and as usual things are crazy. I've finally got my Peanuts experiment up and running, which means people are coming in to participate. A lot of people. The experiment lasts an hour, and between last week and this week, I'll run 24 subjects, which means I'll have lots of data to pour over during winter break.

In the meantime, I also finished coding several strips from Peanuts, and have found several interesting things. For this experiment, I culled 180 strips from the first two Complete Peanuts volumes (kindly donated by Fantagraphics), which were either silent or I altered to become silent. I then coded them all panel-by-panel. That's 720 panels, and yes, it took me all semester.

So, what did I find in my sample? Well, there is some interesting stuff...

Most of what I coded for has to do with narrative structure, or what I would call visual grammar. I'm hoping my redone terminology is transparent enough to follow here.

Out of 180 strips, 140 of them (78%) used "Establishers" to set up information in the first panel. Conversely, 123 of them (68%) finished with a "Release" where the tension of the narrative dissipates. 135 (75%) also use an "Initial" as the second panel, which initiates the actions of the strip. 112 (60%) finish the strip with a "Peak" — the height of narrative tension.

Of 180 strips, 50% (90) use the overall structure of "Establisher-Initial-Peak-Release." The next highest isn't even close, with only 13 strips using the pattern "Establisher-Initial-Initial-Peak."

The "E-I-P-R" pattern is what I think of as the canonical narrative arc (which on a macro scale resembles the traditional "narrative arc" of plotlines). All this aligns even more interestingly to coding I did of event structures for each characters' actions, but describing all that here might be a little overkill.

Just as a reminder, this is a very specific sample of strips and shouldn't be construed as making any sort of claims overall about Peanuts. Nonetheless, its fun to see what info the strips alone hold. Now I'm even more excited to see what the results of my study show about people's behavior in relation to these coded predictions.

New Essay: Japanese Visual Language

For the first time in a long while, I've got a new essay up for download. This one discusses the visual language that underlies manga, and will be part of the Manga: The Essential Reader collection published next year by Continuum Books. Here's the abstract:

Over the past two decades, manga has exploded in readership beyond Japan, and its style has captured the interest of young artists all over. But, what exactly are the properties of this "style" beyond the surface of big eyes and "backward" reading? This paper explores the structural elements of the Japanese Visual Language (JVL) that comprises the "manga style" — ranging from looking at the “big eyes, small mouth” schema as a “standard” dialect, to examining the graphic emblems that form manga’s conventional visual vocabulary. Particular focus will be given to JVL grammar — the system that creates meaning via sequential images — and how it differs from the visual languages from other parts of the world. On the whole, manga provide an excellent forum for understanding the scope of the visual language paradigm.

Capturing vs. Generating Comics

A good friend of mine who works for the company that produces Second Life sends over this link about using the Comic Life software with Second Life screenshots. I've expressed my displeasure with Comic Life before, but I haven't really thought about comic creation of video game clips before.

Something about it rubs me the wrong way... And I think its the same issue that I have with why "photo comics" don't work, and why only some CGI comics feel comfortable.

The problem is that they don't come from some sort of conceptual basis. They are just capturing events in the (virtual-)world and the displaying them in segmented parts. But, contrary to regular comic sequences, they aren't produced to be sequential.

(This may be the same reason that pin-up/cover artists don't always translate to being good "storytellers": they are used to drawing single images, not sequences. Or: they have good visual vocab, not so good visual grammar.)

The capturing vs. generating sequences makes a huge difference, since in one you are actively setting out to express concepts visually, and the other you're just collecting whatever actions might be given to you. In fact, I'm guessing that the CGI comics that read the best (and there are some good ones) are the ones that were first drawn in thumbnails or layouts. The actual "visual language production" occurs at the thumbnail stage. The rest is all just refinement. These "event capturing" comics bypass the stage where visual grammar is deployed.

Of course, the grammar could be deployed "online" in the processs of that CGI comic being created, but I doubt most who do this have much capacity for visual grammar in the first place. They use it thinking that it is an alternative to having graphic fluency, only their non-fluency then shows through in CGI instead of poor drawings.

In many ways this issue is similar to an Internalist vs. Externalist debate in linguistics/philosophy as to where meaning comes from. Traditional philosophy/linguistics (and I think? a commonsense view of meaning?) has held that meaning of sentences is derived from the truth value of how that sentence relates to the "real" world. The Internalist side (including my advisor) says that those meanings only connect to concepts in a person's head, regardless of their truth value to the world.

"Capturing of events" for comics is much like the Externalist viewpoint — sequences of images are depictions of some form of events, and it doesn't matter how they get depicted. The Internalist side would be the opposite: Sequences of images are derived from the conceptual expression of a human mind, and reflect the fluency of that mind.

Video: Visual Grammar

The conference proceedings for the Visual and Iconic Languages Conference from a few weeks ago are now posted online. On the site you can download the slides from my talk, and they've also posted full video of the conference proceedings. I've embedded my talk below, and also to my site.

I highly recommend watching this video of if you are at all interested in my overall theories or in how sequences of images communicate. I describe what exactly I mean by "visual language" fairly clearly, and why it is different from "comics."

Most of the talk though is essentially a snippet of my developing theory of visual grammar — how sequences of images communicate — including my arguments for why panel transitions don't work and my alternative model. I don't plan to post an essay of this work online for awhile (I've been tinkering with it for four years), so this is the best place to find these ideas.

My talk runs for the first 45 minutes of this video, followed by 15 minutes of questions (the second hour is someone else). If you download the slides (large pdf) to flip through at the same time, they might be clearer than on the screen. Enjoy!

The Comic Strip and Film Language

Lacassin, Francis. 1972. The Comic Strip and Film Language. Film Quarterly, 26(1), 11-23.

In this piece translated by David Kunzle, French theorist Francis Lacassin discusses the similarities between the "syntax" of film and comics, noting that they both use "shots" as their base units. For him, this includes various things, like various degrees of framing (long shot, close up, medium shot), dynamic use of what could be multipanel representations ("panning"), as well as semantic alterations, like subjective viewpoints. (I would argue that this isn't "syntax" at all... but that's a larger post).

He argues that though film and comics emerged around the same time, these techniques came first in comics — not the other way around, as is often argued — and that they may have been autonomous developments not influencing each other at all.

He writes: "It is more reasonable to suppose that comic strip and cinema have both separately drawn the elements of their respective languages from the common stock accumulated in the course of the centuries by the plastic and graphic arts." (14)

To this I would question, is it really through historical development, or is this just a reflection of the structuring of people's minds/brains?

He hypothesizes also that film and comics both accomplish their sequential meaning by use of the film theory of montage, which for Lacassin appears to cover most things that do not appear similar to real-world perception.

In his own section at the end, Kunzle criticizes Lacassin for claiming comics were invented before cinema, while those framing techniques are cited being used by authors two generations before that "birth" of comics. Kunzle then discusses the work of 1800s artists Töpffer, Doré, and Busch, noting that they used various techniques like close-ups and polymorphic representation (where one character is repeated in a single frame showing the unfolding of action), among others.

Blasphemy!

As I've mentioned before, I'm currently designing a series of experiments that will use Peanuts strips for stimuli (kindly donated by Fantagraphics). Lately, I've been doing the arduous process of scanning them in and coding them.

Since I'm only trying to look at how sequences of images create meaning, and because the inclusion of text changes things, I've been trying to only use silent strips. Luckily, Peanuts has a lot of those. Additionally, I've also been using strips that have minimal text or could be (gasp!) manipulated to have no text.

I've been taking these strips and deleting the text, then touching up the mouths, etc. so that they work alright silently. It's been quite fun to make sure it all looks like Schulz's style. On the one hand I keep thinking "this is so cool!" and on the other I can't help but think, "Blasphemy!" for defiling the originals.

Most of the time I cut and paste from one panel to another, like putting a frown over an open mouth. In most cases, not much at all is lost in the meaning. It's usually like erasing a word balloon saying "Good Grief!" to just Charlie Brown frowning. The meaning (and humor) stays pretty much the same. I think this is actually a testament to how great Schulz's visual humor was, that I could go in and muck with things, yet the original meanings still come through.

Now... once I start swapping panels around or creating new strips out of panels from a variety of strips, then that might be another story...

(Amusingly, my father actually asked me if there were ethical issues with doing that... my advisor just thought it was pretty amusing. I'd have to think there are less ethical issues involved with retouching comic strips than experimenting on animals like the lab next door, but if I start seeing protests out front I'll reconsider).

Review: The Language of Comics by Mario Saraceni

Saraceni, Mario. 2003. The Language of Comics. New York, NY: Routeledge.

Saraceni's The Language of Comics is one of the few books that attempts to present a holistic theory of how comics work, and draws upon work from "applied linguistics" no less. The book is actually a stripped down version of his dissertation (as is his article "Relatedness" from the Graphic Novel collection). (And good luck finding the dissertation... I had to print it from microfilm on interlibrary loan).

Unfortunately, much of his approach seems to feel of grafting McCloud's work to ideas in applied linguistics in a simplistic (and uncited) way. For instance, he proposes a gradation between semiotic types (like symbols and icons), and can well be compared to McCloud’s Big Triangle.

He treats the sequential aspect of panels as equal to sentences, giving them a “discourse theory” type analysis (like the dissertation by Stainbrook). Saraceni claims meaning is created through commonality between elements in panels, alternating with successive new and given information. He also uses "semantic fields" (connected meanings: like how "snow, caroling, pine trees, and presents" invokes "Christmas") to unite panels not encompassed by this information structure.

However, in doing so, he eschews the role of linear sequentiality, yet provides no argument for why people do indeed read in consistent sequences. The result is essentially a watered down version of McCloud's closure — which it is: his dissertation has the theory in full, and exactly does shoehorn discourse theory onto McCloud's transitions. Some of his insights here are useful and enlightening, yet they deal entirely with "exceptions." He rarely discusses "run of the mill" things like the depictions of events, instead culling his examples from very experimental comics storytelling, like Peter Kuper's The System.

Other chapters cover things like word balloons (perceived as equivalents to direct quotes) and drawings of eyes to buttress a discussion of subjective and objective viewpoints. The final chapter is about computers, which seems out of the blue and has next to nothing to do with comics.

Since it uses applied linguistics, much of the book feels attuned to what might be useful for literary studies. To this extant it might work very well. However, as a theory of "meaning" it falls short, largely because it does not address any type of cognitive system, and lacks even McCloud's precision of surface categorization.

My biggest gripe about the book is that it is presented as an introductory textbook as part of the Intertext textbook series, has no citations outside a "recommended reading" list in the end, and is written with a matter of fact tone that presents it as an authoritative stance on the body of knowledge of this field. The truth is that no such body of knowledge exists at this point for "comics theory." Right now, we're in that period of science where lots of different viewpoints are popping up, just waiting for an encompassing paradigm shift to sweep in and take over.

Even if I were to come out with a book of my full theories, it wouldn't be a de facto textbook because it would be my views drawing upon that body of research. As a result, to those "in the know," the format and style make this book seem misleading in its intents for fronting Saraceni's views as well established scholarship.

To end on a good note: Though I think it fails in not using a cognitive approach, I do like that the book tries to use concepts from linguistics with "comics." It shows that this type of approach is not just intuitive to me, but to others as well, and locating the book in the broader field of linguisics is good for the field as a whole.

Coercion... of meaning!

Today I gave my big first year project presentation to the psychology department. From what everyone has said, it went very well. Of course, the project itself is still underway, and I will be running several more subjects in the lab, while probably continuing the study as a whole throught the summer. This shot is of me and my advisors from afterwards. (R to L: Ray Jackendoff, Phil Holcomb, Me, Gina Kuperberg):

As I've mentioned before, my talk involved looking at the "Event Related Potentials" or ERPs involved with processing a certain type of linguistic phenomenon called "semantic coercion." ERPs are a measure of the electrical activity of the brain. We don't get a good fix on specific brain areas that are at work, like in fMRI, but we do get very detailed analysis of the time course of events and certain waveforms do seem to indicate types of brain functioning in contrast to each other. We measure this electrical activity by sticking a cap of electrodes on people and feeding the signals into a computer, which then averages out the noise over several subjects and trials to give a smooth wave for time locked events. Here's me in the cap...

So, I looked at these brain waves for semantic coercion, which involves the extraction of "hidden" meaning from sentences like The chef finished the chicken before the main course. Someone can't literally "finish a chicken," they have to finish doing some action with it, like cooking. Since the event isn't stated outright, it's said to be "coerced" from the combination of the verb "finished" and the direct object "the chicken." Here's a waveform from one of the sites on that cap that I got in the experiment:

While this is interesting as a linguistic phenomenon, I think it's really just a warm up for more comic related studies. Since I couldn't resist, I even opened my talk by showing this strip:

Now, if you look carefully, coercion happens here too. We never see the event of Snoopy catching the ball, yet we know the event happens based on the information provided by the other panels. In addition to other things, coercion is perhaps one of the things that McCloud was trying to get at with his notion of "closure." In many ways, coercion here is an invisible meaning that is created out of the visible components of the graphic sequence. Graphically, it's the stuff that happens "out of view" of the panels. The problem is that McCloud extended this to the (linear) relationships between all image sequences, which just doesn't work.

So, if I do find anything fairly robust in the ERPs for verbal coercion, perhaps a study of visual language coercion could be on the horizon as well? Or perhaps a theoretical paper first...

You're a good grammatical construction, Charlie Brown

I've recently been thinking quite a lot about how best to start testing my theories of visual language grammar. Since I'm in a psychology program, I've got to actually think of experiments that might yield reliable and significant results (hopefully).

One of the main ideas I had was starting off using a corpus of comic strips, so I wouldn't be biasing the study with my own drawings. I hit on the thought that Peanuts strips would be perfect for this since 1) there are a ton of wordless ones, 2) they're well recognized culturally, and 3) they use a fairly simple bare bones structure with 4) nearly always with 4 panels.

So, thanks to a very kind donation from Fantagraphics, I am now pouring through several volumes of The Complete Peanuts strips in search of all the wordless/minimal text ones I can find (there are a lot!). Hopefully, by summer I should be testing peoples intutions on the grammar of these strips, and eventually looking at their brainwaves while processing them (fun!).

One of the things that has jumped out at me is how so many of the strips use systematic patterns that I haven't noticed before. Previously, I've talked about the visual grammatical pattern of the 'Set up - Beat - Punchline' construction (as coined by Neal VonFlue). This is the pattern that sets up the joke with dialogue, then has a pause panel, then ends with the punchline. Well, Schulz seems to use a few other patterns a lot as well.

The most intruiging to me is one that is almost exactly like the SBP pattern, only the "beat/pause" panel isn't actually a pause: it's an "action" panel (SAP?). Instead of a passive type "rest," the space is filled by some wordless action that sets up the payoff with the final panel punchline. I've only looked at the oldest of the collections (the 1950s) and have only seen a few actual SBP constructions. I'm curious whether or not this SAP pattern preceded/led to the SBP one.

Another pattern has the first three panels as wordless depictions of an event, only to have a final panel with a punchline that explains or comments on the actions. This one happens extremely frequently, and sometimes takes on an additional characteristic of having the first panel depicting an action as well. It starts with an event that sets up the primary event that unfolds in the rest of the panels.

Patterns like this are fun to find, but can also be challenging theoretically. At least as far as developing a model for my visual grammar, sometimes I'm hesitant of how to notate certain panels, and often debate which is more correct. Imagine not only trying how best to describe how nouns and verbs combine, but also whether or not things are nouns and verbs in the first place and/or whether those categories are appropriate at all (when there's good evidence for all).

And, unlike with homework, there is often no answer key that I can check with someone else (except, hopefully, what my experiments will reveal). I've always found this "working without a net" to be a little scary, but at the same time exciting since it portends new and uncharted territory. I suppose it's the feeling of truly doing science instead of just learning it.


Note: As long as I'm giving thanks for donations, I should also mention the kind contributions of TopShelf, Drawn & Quarterly, Top Cow, Oni Press, and Dark Horse Comics. Their generosity will make a huge difference in these visual language studies and are greatly appreciated!! If you are from another company and would like to donate to this cause, please contact me...

Dunkin' pictograms

Speaking of pictographic writing systems that will never be universal, Fabricari sends along this logo from Dunkin' Donuts:



I've been seeing that logo for over a year now, and never quite got that the images were supposed to be a pictographic sentence until a few weeks ago! I had just thought that it was a bunch of pretty pictures. Of course, given that I rarely if ever go to Dunkin' Donuts, I never really put in the effort to try and decode it.

Though, while we're here, I might as well use it as an example as to why pictographs fail as universal systems. Outright, as I mentioned in my last post, the grammar here completely mimics that of English. Of course, that was the intent in this case since its a slogan, which is why there is four units for four words. But, notice also that this conversion makes a very important decision: it chooses to transcribe "America runs on DD" as a verbal-->visual mapping, rather than siphoning the concept behind the idea into the visual modality to then adapt to its own traits. Moving on...

Let's start with nouns. First off, the DD is only understandable if you know the association to the company. The map of America could be read as "map" or some such, but is even more interesting since it is a metonymy. It uses "America" to mean "the people in America."

The verb "run" nicely shows how you can't visually show an action without also showing an object. It's tough to show "run" without also showing the "runner." Verbal grammar (by virtue of its symbolic nature) likes to divide these pieces into [ACTOR]-[ACTION]. In visual grammar this division doesn't work as well (being iconic, not symbolic), instead becoming [ACTOR:state1]-[ACTOR:state2], where "state2" shows the fruition of the action.

Finally, the preposition "on" isn't visually converted at all. I find this to be particularly telling, since it immediately shows the English context. I imagine also that the makers of the logo struggled with it, since this usage of "on" is not the spatial kind ("on top of") but is tied to the verb.

In fact, the interpretation of "run" as an action here (like "run down the street") is wholly off, since they don't mean that Americans "use their legs to run on top of Dunkin' Donuts." Rather, they are using a construction "run on" (arguably not two units) that means roughly "to be powered by." The "person running" image then becomes a "double rebus" --> first mapping the sound pattern to the image, then the image's literal meaning to it's "metaphorical" meaning.

To come back around to my initial statement about mimicing English grammar, this actually can't even do that since the slogan doesn't use concrete elements. A literal reading of this ends up being totally bizarre (bracketed by panel, italics adding clarifying info):

[The country of AMERICA][uses its legs to RUN][ON top of][DUNKIN' DONUTS]

SO... this example only goes to reinforce how hard it is to "accurately" map verbal expressions to visual signs — both individual signs and grammatical sequences — especially when it involves metonymic and metaphorical expressions (add those semantic aspects to the list then). While English speakers might be able to figure this one out, can anyone possibly imagine this working on a global scale?

Time essay analyzed

In Derik's continuing exploration of panel transitions (Part 1, Part 2, Part 3), he does an interesting job of dissecting my latest essay "Time Frames... Or Not." To keep things localized, I'll make my responses there, but he seems to have done a fairly thourough job of it. Worth perusing.

Also, Blogger has helpfully decided to add tags in finally, so I'll be trying to work those into all my past posts in due time.

Problems with Transitions

Over at Derik's blog he's been examining McCloud's panel transitions based on influence from film theory (Part 1, Part 2, Part 3, Part 4, Part 5 ...more to come).

While Derik only does it a little bit, the application of film theory to panel transitions isn't altogether new. John Barber essentially grafted McCloud's and my own (old model of) transitions onto Eisenstein's thesis/antithesis/synthesis model in his masters thesis. This was then argued against by Ben Woo in his thesis, dismissing it more because modern film theory does than any explicit argument against Barber's thesis. I'm not up on my modern film theory that much, but I believe Eisenstein is fairly passé at this point anyhow.

A few months ago I started noticing how similar Eisenstein's montage was to the cognitive linguistics notion of "Blending." Blending takes two concepts and extracts parts from them to create a new entailment. A classic example is "The surgeon was a butcher" — both surgeons and butchers are skilled at cutting flesh/meat, yet when combined together they illicit a meaning that the surgeon was sloppy. This is just like the 1+1=3 idea from montage.

And it certainly does appear across panels. I had a whole section on blending in my paper A Force of Change. Though, I think that the structures governing sequential understanding (i.e. syntax and semantics) are different from this.

Really, Eisenstein's montage and McCloud's closure are kind of like the film/comics equivalent of ether; a magical "mental" substance that doesn't really exist that glosses over any real substance the mind might actualy be contributing. They're like pop-science: a simple easy explanation for a very complex phenomenon. Just like Freud and Jung are still thought of by laypeople as being what psychology is about, their theories are far left behind to modern thinking. In fact, I'd venture to say they're more used by humanities/social sciences these days than psychology or cognitive science.

Of course, I've been railing on the panel transition approach for quite a while now, over the course of several alternative models. And, it's not just the idea of transitions that has problems: it's any approach that only takes into account panels that are immediately adjacent to each other. Any linear approach to the idea of creating meaning in sequential images will ultimately fail.

As I mentioned on one of Derik's posts, the major shift comes in what one is looking at. Instead of looking at panels' immediate surroundings and basing the system around those juxtapositions, we can instead acknowledge that whole sequences mean things (events/actions/situations/ideas). From there, it becomes a matter of identifying what functions different panels play in creating that overall meaning. Just because we read and write panels linearly doesn't mean that's how we understand them.

Nevertheless, it's interesting to watch Derik go through steps in his thinking in relation to what I did. He named it "rethinking transitions" so it'll be fun to see what his rethinking leads to.

Updated 12/1 with additional links to further entries

Tim and Time

So, well timed for my new essay, "Time Frames... or Not", on why panels don't equal moments (and time does not equal space), Tim Godek posts this excellent and simple example of a temporal paradox. (I'm not reposting it here because the file size is rather large, so go look yourself!)

Since the three "moments" of the event happen across the same background, a "temporal map" reading would force the foreground figure to be hovering in front of the background, or, the background shifting behind the character. I think that we're forced to reconcile that the person is doing an action that occupies a singular space (i.e. sitting and thinking) while the background does not remain consistent behind that static foreground space. So, either fore/background is shifting or it creates a paradox of temporal progression where we parse the foreground figure in his own "conceptual space" separate from the background.

This also relates back to the "positive/active" versus "negative/passive" elements I talked about in my "A Visual Lexicon" paper. I think what makes this strip work without a "shifting" interpretation is that the Tim character is Active in the sequence compared with the Passive background. So, our focus of attention is held on him rather than on the consistency and oddness of the background.

In either reading, there is something in the content that illicits a "not normal" reading (i.e. moving when sitting vs. static yet background conflict). If anyone finds more of these temporal paradox type examples, definitely send them my way.

New Essay: Time Frames... Or Not

Wow, its been a really long time since I last posted a new downloadable essay. Well, if you've been anxiously awaiting one, today is your lucky day! I've just posted my latest theoretical offering, "Time Frames... Or Not," where I tackle the assumptions that lead to the (false) belief that successive panels equal moments in time. Here's the full abstract:

The juxtaposition of two images often produces the illusory sense of time passing, as found in the visual language used in modern comic books. While this linear sequence may seem on the surface to present a succession of individual moments, the understanding of graphic narrative is hardly so simple. This paper will explore how the linearity of reading panels and the iconicity of images create various assumptions about the conveyance of meaning across sequential images relation to space and time.

Astute and long-time readers of this blog will remember that I mentioned writing this paper way back in January of this year. Its good to finally get it done and out!

Like many, I'll be back to school come Tuesday, so hope you all enjoy the day off!

San Diego, coming soon!!

Thanks go to Scott McCloud for kindly linking to my latest Comixpedia article on "visual rhyming." He suggests "Eye Rhymes" as another good term, and I agree, it does sound pretty cool.

He also notes that he'll be speaking at the San Diego ComicCon on Friday from 12-1pm. I know I'll be there, and so should you: Scott gives great talks. And immediately after Scott's presentation, you should walk over and see me talk at 1:15 to 2:30 on the Visual Language panel of the Comic Arts Conference!

This year's panel should be fantastic. I can't wait, because my talk will be the best presentation I've ever done. I don't want to overhype it, but I've been working really hard on it and it should be great. Called "The Secret of Sequence," I will finally be unveiling most of the my model of Visual Language Grammar, so, if you're curious at all about "how sequential images create meaning," then you won't want to miss it.

The panel will also have two other talks about theory related things as well. I'll post more about them, and the location, as the date nears.

Thoughts on Time

I've had a couple very interesting (and heated) discussions about "time" with friends recently that might be interesting to share. What had inspired one of these discussions was a book I'd read that implied that human's ability for narrative gave way to ideas of past, present, and future, and thus a sense of time that is unavailable to animals.

My friend, ever the philosophical champion of animals (bless his heart) responded that this isn't the case at all. He noted this example: his cat can watch its prey go behind an object and will know its going to come out the other side, so it goes to the other side to trap it when it comes out. So, since it can predict the result from a starting action, it has a sense of a future, and thus a sense of time.

I disagreed with this categorization, thinking that a conception of "time" as a thing is different from the cause and effect knowledge of events in daily life. He replied that it's just a matter of degree, not kind.

My thoughts right now are that there are two things going on here:

1. The subjective experience of time through the events that happen through during the day.
2. An abstract sense of time as something objective and outside just experience.

Whether it’s a matter of degree or kind, some type of symbolic activity (be it language or otherwise) is necessary in order to move from a conception of the first type to the second.

So, what does this have to do with visual language?

Well, many people believe that panels somehow equate to moments, and that "movement through space is movement through time." I believe that the second belief in time underlies these assumptions. The whole sequence of panels (or just physical space) is looked at as an abstract passage of time in which the actual panels (or internal parts of panels) are seen only as parts that fulfil the expectations of the broader whole.

In contrast, a view of panels akin to #1 focuses only on panels as segments of events. That is, the understanding of panels in sequence is far closer to the experiential sense of events than to an overarching sense of time (as I suggested way back in my Buddhism essay).

Timing

Newsrama hosts the first part of three articles about Time in sequential art written by Joanna Estep. The piece is very well presented, and I like how systematic her analysis is, especially her use of diagrams to push along the theory. It's well worth reading, and I look forward to seeing what her next installments bring.

However, I also want to point out that it makes certain assumptions that are largely passed on from the Eisner/McCloud tradition. Mainly, it holds that "one panel = one moment," which simply isn't the case if you actually look at sequences of images from books (as opposed to just mental theorizing – of which I've been guilty of too). There is nothing about two panels that dictates time is passing – only content that implies temporal succession can yield this result. And, once you see that many panel sequences don't inherently push time along, you realize that problems arise in any linear notions of time across panels.

Following this, it also reinforces the ideas that "spatial distance = temporal distance." I had some thoughts on this like four years ago that I've never really worked into a full-blown paper, but the basic idea is that panel sizes create a rhythmic structure for reading. To really see if this is true I'd need to do eye-tracking studies though…

I'll hopefully be posting an essay I've been working on about Time myself sometime soon, but till then my old essay Visual Syntactic Structures (and book Early Writings...) delves into these things for anyone interested.

Update: I now see that Timing Part 2 is posted too. Again, worth reading, but continues the assuptions in McCloud that "reading time = fictitous (i.e. mental) time." I'm also curious why she includes her "hierarchies within images" as being related to time, since she doesn't measure any increase or decrease thereof. I agree with this: I don't think foregrounding is related to time at all, though I do think its related to distinguishing things like who is the focused actor and who is subsidiary.

Update #2: Timing Part 3