5 Card Nancy and Panel Transitions

One of Scott McCloud's more wacky inventions is the game Five Card Nancy which is based on the old comic strip Nancy. The basic premise of the game is that you can create lots of different (and fun) novel strips by combining random panels together. Scott recently posted an old collage he did that led to the game.

Of immediate note in his collage is that the sequence doesn't exactly make much sense, despite some cohesion between the panels. I'd say that it may have a narrative structure (i.e. visual grammar), but no meaning (semantics).

In some cases though, the juxtaposed panels do make sense, but the global meaning does not. In linguistics (borrowed from math), we'd call this a "first-order Markov chain", since only the units right next to each other have a connection. If a panel had a connection to two panels next to it, it'd become a "second-order chain", etc...

Markov chains were the primary way that people thought about language's grammar up until the 1950s, when Noam Chomsky then showed that grammar needed to account for connections farther than just countable individual word relationships (an approach I then applied to comics' sequences).

Essentially, McCloud's theory of panel relationships is a first-order Markov chain theory. It only looks at juxtaposed relationships. Interestingly, his Five Card Nancy game follows the same characteristic. Since players put down one panel at a time, it appears as though they are just making choices linearly. However, I'm guessing that the higher scoring combos are all ones that gel on a global scale, not just a local connection.

Also, the limitation of the panel transition viewpoint is really highlighted by McCloud's Nancy collage. How can panel transitions be correct if only local connections make sense but ones further down the sequence do not? Though we may draw and read comics one panel at time, it doesn't mean we don't build or project a bigger structure in our minds beyond the linear relations.

Comics reading: Competence and performance

Often times when I give talks, especially concerning layout or the visual grammar of sequences of images, one of the questions inevitably says something along the lines of "But, there's no guarantee that the reader will view a page in the proper order." The high variability of possible choices or readings of a comic page makes it hard for them to accept a steadfast theory of comprehension.

However, a similar issue was at play in linguistics back in the 1950s, and was one that Noam Chomsky importantly addressed in his distinction between competence and performance. Competence refers to the (idealized) organization of rules and constraints in our minds that guides us to understand language. Performance is the vast variability that happens in real life exchanges.

For example, someone might say something like this over the phone:

I ...uh... I went *cough cough* to the store *STATIC***--oday and, like, ...um... saw *CAR HORN*--ohn from my class in the check-*hiccup*-out line.

There are lots of interruptions, unclear portions and distractions. However, most likely a listener would glean from this a sentence like:

I went to the store today and saw John from my class in the checkout line.

The rules in your head are not bothered by the messiness of the context — your attentional system can filter out a lot of it.

The same is true of reading a comic page. Let's say you start in one panel and go to another, then realize it shouldn't have gone next. You're not belying the mental rules that go into comprehension — in fact, those rules are what tell you it's the wrong order. These actual rules of comprehension are unconscious to your awareness.

Your (unconscious) competence wins out over the messiness and variability of performance.

This same issue may be at play with comparisons of comics to film. Yes, film and comics are presented differently (one static, one moving), but that doesn't necessarily mean that their comprehension in people's minds is entirely different. The difference in presentation may be a "performance" issue, while the comprehension is a "competence" issue. (Though, in my mind there is bound to be at least some variance due to that presentation difference — motion vs. static — which will need experiments to explore... yay science!).

I should point out also that, in linguistics, there are some debates over the complete reality of this split in notions. For example, for a long time it was argued that words like "um" and "uh" are just performance clutter. However, research has shown that these actually hold meaning for the discourse (essentially signaling how long a pause the speaker is going to make before continuing to talk).

Nevertheless, for many issues facing the comprehension of "comics" (and/or film), it is an important split to make.

Action!

In discussing this post of mine with Derik, I realized that I should post on the technique I used of substituting a whole panel for an "action star," like this:



This usage is somewhat similar to what I talk about in this older article on metonymy, and the same phenomena creeps up overtly in McCloud's famous "Closure" example from Understanding Comics:



In both cases, we never see the action, because it's replaced by a panel that implies action took place, but replaces the image with some neutral information. In the case of the action star though, we associate that sign with events, so it further indicates the presence of an action, whereas in McCloud's example the text does most of this work, since the cityscape is entirely neutral.

I've recently been exploring this phenomena a lot more, especially since I keep seeing it in Peanuts strips, first shown (I believe) in this one:



So, we now have this phenomenon where we know we can substitute certain types of panels for others to get an entailment of the actions. For storytelling, this is pretty cool, since it forces the reader to draw an inference about the actions (a result interesting enough that McCloud extended this out to all interactions between panels).

However, are there also restrictions on which types of panels we can replace? Since the action star essentially just means "events occurring!" but doesn't show them, it can be considered as a kind of "visual pronoun." Because of this, it can also be used as a diagnostic for determining certain categories of panels versus others. This "pro-form" replacement is a common technique in linguistics for determining grammatical categories: we can replace Noun Phrases with the pronoun "it", and Prepositional Phrases with "there":

1. Martin pushed the really huge boulder up a massive hill.
2. Martin pushed [it] up a massive hill.
3. Martin pushed the really huge boulder [there].
4. * Martin pushed the really huge boulder [it].
5. *Martin pushed [there] up a massive hill.

In 2 and 3, we can see that this substitution works fine, but when we reverse which ones we're substituting for, in 4 and 5, it sounds awful (indicated by the asterisks).

So... can we do this using an "action star" as a kind of visual "pronoun"? Check out these Peanuts strips, where the action star replaces either the second or third panels:

1. *
2. *
3.
4.

When the second panel is replaced, it does not seem to make much sense (nor would it make much sense in the first or final positions here either). However, replacing it for the third panels does work — hinting that those panels belong to a certain class of words where a culmination of an action occurs (even when that action isn't an "impact").

Note also that an approach using linear "transitions" between panels would be unable to express this: what would the action star be a transition of — a "Non-sequitur"? That wouldn't be able to capture the understanding of the event occurring in that panel. Rather, this hints that transitions (based on the relations between panels) are not the way sequences are understood, and the need of some sort of global narrative structure (with categories for actual panels) underlying the sequence.

The Development of Language

This related to both sequential images and linguistics, so how could I not post it? Via LanguageLog:

Equivalences for "Language"

In claiming that the graphic form (especially in sequence) is structured as a language, it might help to parse out how to make the analogy reasonably. It's not as if no one has ever noted the similarities between the forms — in fact, it's fairly common. However, mistakes are often made in my opinion.

First, I assume that there is "Equivalence" between the different modalities, which can be summarized as "the expectation that the human mind/brain treats all modalities in an equal way, given modality specific constraints."

By this account, it essentially means that we would expect all modalities to feature the same sort of storage in memory for patterned signs of differing sizes and functions (from sound patterns of phonemes to sentence patterns of idioms) and feature ways to combine all those elements at different levels of structure (phonology through discourse). We would also expect development to be similar, with a critical learning period and drop offs after that.

However, this is not the take that most comparisons of the verbals and visual forms take. Rather, they often try to make direct superficial analogies between specific types of structures. For example, "such and such" is the equivalent of a "word" or "sentence." This is often why many want to claim that single images have "grammar" — because a single image has lots of information in it, like a "sentence" and unlike a "word" — even though composition within single images behaves nothing like a grammar. (...nor should we expect it to given the differences between sound and light!)

A similar endeavor has tried to find "minimal units" of the structure of the forms, following the school of Structuralism (most popular in American linguistics from around 1920-1960ish). However, again, just knowing minimal units doesn't tell you about the broader structure, and units larger than minimal units might also be useful and insightful. It also gives no beneficial comparison other than that "minimal units" exist in both domains.

All of this is an argument for looking beyond the superficial understandings of "language" and to look for comparisons in deeper, more fundamental aspects of structuring.

Fun with Text

While preparing the Peanut strips for my next project, I came across a fantastic integration of text subtly hidden in this panel:



If you look carefully, at the center of the starry smack mark where the ball hit the bat, there is text reading ".315", which I assume is a reference to Pig-Pen's batting average (pretty good). This is particularly interesting to me since it's a descriptive use of text as opposed to a sound effect.

xkcd has a similar usage where instead of sound effects, the text reads the name of the action being done. However, the result in xkcd is that it describes the actions straight-out.



These two types of usage give complementary aspects of the way languages structure actions and manner of motion. Compare:

a. The ball flew into the glove.
b. The ball spiraled into the glove.

In sentence (a) you are given the action that the ball does, but little about the characteristics of that action. Sentence (b) gives you the manner of the motion, from which you derive the action implicitly.

Sound effects often give you information about manner of motion. For instance, a golf ball falling into a hole goes "Klunk" or "zoom" to describe a speeding car. These elaborate on the action itself. The use of text in xkcd eschews this to just focus on the action, without any manner of motion. What is intriguing about the Peanuts example is that ".315" is neither manner nor action — it is purely descriptive in an additive sense.

Playing with this one step further, we can create some panel pairs that replace the action for the sound effect. One characteristic of these types of "action text" is that they can stand in for the actions themselves (discussed a bit in Interfaces and Interactions). Note that both replacing for an action or manner of motion works fine:




However, substituting ".315" is a little weird — even with the expectation of the event — since it doesn't stand in for the action itself. It only gives you additional information about the action:



Looking through all these Peanuts strips, Schulz was more of a formalist than he's thought of I think.

Podcast: "Grammar" in visual language

I've done another podcast with the folks at VizThink, this time debating Yuri Engelhardt and Dave Gray on what constitutes a visual language and the nature of visual language grammar.

This new format allows you to skip around to different chapters to jump straight to parts of interest. (Please note, I object to the insinuation in the chapter title that "comics" can equal "visual language"):

Hint: Use the Full Screen Button to see this video in greater detail.

I think that there is something I strived to point out throughout the discussion that I didn't articulate well enough, but to explain it I'll have to do a mini-linguistics lesson.

In the podcast, Yuri pointed out the view that language has two main parts: a set of units (lexicon) and a set of combinatorial rules (grammar). This view of two components is essentially Chomsky's view of grammar, and organizationally looks something like the diagram to the side.



In this traditional view, syntax/grammar is the component that offshoots meaning, and only syntax has properties for combining elements together. I said that I agreed with this notion, but really I don't. When I mentioned that I subscribe to a view from Chomsky's student, Ray Jackendoff (my teacher), I should perhaps have elaborated on the differences between those perceptions more, because they are extremely important and can resolve some of the conflict of the debate.

Jackendoff's view of grammar is different. This "Parallel Architecture" says that the mind has three main interfacing components: modality (auditory/manual/graphic), syntax, and conceptual structure (meaning). The "lexicon" is distributed across the interfaces between all three of these structures — it doesn't have it's own "place." And, importantly, each of these structures has that capacity for infinite combinations — not just syntax. (Note the similarities to my listing of properties of Language). This would look like this:



Much of our debate focused around whether or not single images (diagrams) have "grammar." My objection is that it does not function like "syntax" does in a verbal grammar, though I acknowledge that there might be a hierarchy or a combinatorial system there. If you subscribe to the Chomksyan view of grammar, you're forced to say that the combinatorial element "is syntax," which is exactly what Yuri is doing:



If you follow the Parallel Architecture (as I do), syntax is not the only element that creates hierarchies. They all do. So, combinations within a single image or diagram is "grammar' insofar as phonology is the "grammar" of sound. Essentially, Yuri's "visual grammar" is the combination system within the graphic structures, which is why I kept prodding about the difference between it and just the system of perception (and why most of its "constraints" are based on iconicity). This instead looks like this:



In contrast, my grammar for visual language needs a combinatorial system for individual images and for combining them together, looking like this:



To the extant that the narrative structure takes concepts and a modality and orders them coherently, it functions the same as syntax in verbal language. This is "visual language grammar" analogous to the way that syntax is verbal language grammar (nouns and verbs). But, all three structures have combinatorial properties. They don't all make reasonable analogies to saying that they are like "grammar" in the syntactic sense, but they may be combinatorial.

(This is also why you can say that "gestures are to sign language what individual images are to visual language" in the context of sequential images, but not for individual images. There is no developmental/fluency gap like this for "... visual objects are to individual images". I.e. People don't learn how to draw simple graphic signs but not be able to put them into a diagrammatic arrangement.)

Making this shift in perception buys you a lot: It makes the distinction why single images may have hierarchy (like perception/phonology), but don't have grammar (like syntax). It addresses why most of that structure is guided by iconic and indexical constraints. And, it also may give you a leg up in describing combinatorial aspects of images beyond diagrams (which occurs within panels).

Finally, it is worth noting that not just aspects of language have consistent patterned units that appear hierarchic in structure within our cognitive system. This also appears in music, event structure, vision, social structure, and a myriad of other domains (discussed well here). But, we don't have to call them "languages" because of this broad similarity.

Suggested reading:
Foundations of Language and
Language, Consciousness, Culture, both by Ray Jackendoff

Thoughts on Language Evolution

In my TA class this semester we've just entered talking about Language Evolution, and combined with the recent discovery of this blog the topic has been on my mind a bit lately. Some general thoughts on reframing the overall discussion...

Some theories of language evolution postulate that ‘gestural language’ evolved prior to verbal language. While I am in support of the multimodal sentiment, parts of this rub me the wrong way.

First off, humans currently use both modalities concurrently in expression, which is offered as part of the evidence for its potential importance in evolutionary contexts. While the verbal form in most people uses more complex structure, both forms are in use at once in co-speech gesture. Why would it make sense for only one modality to develop dominantly then transfer into another? Should we perhaps be thinking in terms of concurrent development for concurrent usage?

Along these lines, humans also have the capacity to draw. While our primate cousins do have gestures and vocalizations for conceptual expression, none seem to manipulate the world for conceptual intent, which is at the heart of drawing. Yet, this is never mentioned alongside the discussions of the other modalities (though not surprisingly).

Also, I think it is largely a misnomer to say “gestural language” evolved first. As a cover phrase, I think it obfuscates the issues involved — namely, that there are various mental structures that contribute to a behavior of manual (verbal and visual) expressions.

This is even more problematic when various people use the word “language” in different ways — some refer to a communicative system, some to grammar, some to conceptual expression. Really, if the discussion is about the evolution of what we now know of as “language”, it may indeed be inappropriate to talk about its historical states as if they're the same thing as we have now.

It would be more useful to discuss the development and evolution of these structures (syntax, semantics, etc) than to talk about the capacity as if it were a whole thing. That is, we should be talking less about the evolution of “Language” and more about the development of the interacting cognitive structures that end up contributing to language as we know it in the modern context.

Art, Language, and "Cognitive Equivalence"

When I usually speak about the Art versus Language Perspectives, I usually couch it in a view that there are "different potential ways our society treats graphic images." As I just realized, stating it in this way maybe obscures the true intent of the distinctions.

Really, this is a hypothesis about cognition.

At the heart of my theory of visual language is the observation that we have three modalities by which we can convey our concepts: Sound, Body motions (faces/hands), and creating graphic images. That's it. To push it further, the theory is that when these modalities take on structured sequences governed by a system of constraints (a grammar), it becomes a type of language: speech, sign language, or visual language.

The "Language Perspective" assumes that all systems of conceptual expression work in similar ways — what we can maybe call "cognitive modality equivalence" or some such. Under such a view, we would expect for the graphic modality (drawings) to operate under the same principles as the verbal and manual domains.

If you look at the ways that speech and gestures (and sign language) grow developmentally in children and are used and treated in society, you see certain patterns — conventionality, imitation, communality, etc. While many of those patterns do emerge in the graphic domain, they appear "dampened", are dismissed, or just aren't recognized as such.

So, the question becomes raised: "Why don't you see these things fully in the graphic modality?" and/or "why don't we know them when we see them?"

The offered explanation is the Art Perspective — a cultural force that suppresses the patterns that would normally emerge from any other modality of conceptual expression. With polar opposite emphases, the Art Perspective works to dampen the "usual" course of development and treatment for graphic images.

So, given this, a new set of questions can be asked about this underlying "cognitive modality equivalence": What are the trends that a conceptually expressive system shows in development and society? How do modalities differ? Do these trends reflect broader cognitive processes than just conceptually expressing systems?

Don't!

My department here at Tufts is hosting a big linguistics/psychology conference this weekend that everybody is going crazy organizing and getting ready for. I'm not really doing that much though, since I'm not presenting or participating that much (I'm actually heading to Santa Barbara on the weekend to watch my brother graduate college!).

Anyhow, my advisor asked me to draw a few comics for his upcoming talk and book so I figured I'd share. I'm not sure what the full discussion will be, but he said these will be used to talk about how sometimes reference occurs in places outside of sentences themselves:

Coercion... of meaning!

Today I gave my big first year project presentation to the psychology department. From what everyone has said, it went very well. Of course, the project itself is still underway, and I will be running several more subjects in the lab, while probably continuing the study as a whole throught the summer. This shot is of me and my advisors from afterwards. (R to L: Ray Jackendoff, Phil Holcomb, Me, Gina Kuperberg):

As I've mentioned before, my talk involved looking at the "Event Related Potentials" or ERPs involved with processing a certain type of linguistic phenomenon called "semantic coercion." ERPs are a measure of the electrical activity of the brain. We don't get a good fix on specific brain areas that are at work, like in fMRI, but we do get very detailed analysis of the time course of events and certain waveforms do seem to indicate types of brain functioning in contrast to each other. We measure this electrical activity by sticking a cap of electrodes on people and feeding the signals into a computer, which then averages out the noise over several subjects and trials to give a smooth wave for time locked events. Here's me in the cap...

So, I looked at these brain waves for semantic coercion, which involves the extraction of "hidden" meaning from sentences like The chef finished the chicken before the main course. Someone can't literally "finish a chicken," they have to finish doing some action with it, like cooking. Since the event isn't stated outright, it's said to be "coerced" from the combination of the verb "finished" and the direct object "the chicken." Here's a waveform from one of the sites on that cap that I got in the experiment:

While this is interesting as a linguistic phenomenon, I think it's really just a warm up for more comic related studies. Since I couldn't resist, I even opened my talk by showing this strip:

Now, if you look carefully, coercion happens here too. We never see the event of Snoopy catching the ball, yet we know the event happens based on the information provided by the other panels. In addition to other things, coercion is perhaps one of the things that McCloud was trying to get at with his notion of "closure." In many ways, coercion here is an invisible meaning that is created out of the visible components of the graphic sequence. Graphically, it's the stuff that happens "out of view" of the panels. The problem is that McCloud extended this to the (linear) relationships between all image sequences, which just doesn't work.

So, if I do find anything fairly robust in the ERPs for verbal coercion, perhaps a study of visual language coercion could be on the horizon as well? Or perhaps a theoretical paper first...

16 panels that are Still Conventional whether you think they work or not

I neglected posting this ealier in the week, but John Morris has a parody/homage to Wally Wood's 22 Panels that always work over at Comixpedia entitled 16 Panels That I Don't Think Work All That Well. There are a couple things I find theoretically interesting about it.

First off, it is a great compilation of conventionalized patterns used in many comics. Just like WW's 22 Panels and the Peanuts patterns I've been finding recently, this list excellently shows how there are systematic and conventionalized patterns in the visual language used in comics. This is in contrast to the view that graphic creation is unrestrained because it mimics perception, and thereby lacks an experss system of mentally stored patterns. Astute observations like these 16 panels excellently show that there is systematic and patterned visual vocabulary used by "visual language speakers" (and I wish more people would do work like this!).

The second thing this list shows is a preference for the use of some representations over others (WW's 22 Panels does this too, though positively as opposed to negatively). Linguistics has often been in perpetual debate with journalists/English teachers/etc. that believe there is a right or wrong way to use language. We are often told "not to end a sentence with a preposition," or "not to split infinitives" — though these are not in any way real rules of English grammar. (They were prescribed by grammar book writers in the 18th century who were enamored with Latin — so they advocated Latin's rules for English, not at all being sensitive to the fact that, y'know, they're totally different languages!!).

This list's intent is prescriptive in the same way. Despite these being consistent conventional trends used in this visual language, they are gauged by their value in usage. An additional aspect to this is the Art perspective most invoked in the comments below the article. Most people object to these panels simply because they are conventional! They're "overused," which means they aren't new and innovative/original — which makes them undesirable to the Art viewpoint.

However, none of this mitigates the fact that these panel types are conventional. The linguist would say "they're all part of language, let's observe how people use them" while the prescriptivist says "they're part of language, but they really shouldn't be, and those who use them are less sophisticated speakers."

It's interesting to note also that no matter how loudly prescriptivism might object to such "bad" usage, it never has an effect on shaping language usage. It's not like split infinitives have gone away because people advocate against them! Nor do I suspect these 16 panels to go away either.

In some ways both aspects of a list like this shows some good headway in recognition of this visual language as a language on the whole. Not only are people recognizing the patterns, they're also judging them prescriptively, just like any other language!

Speaking fun

As I mentioned in a previous post, I'm going to be speaking next week at the Popular Cultural Association National Conference here in Boston next week. If you're interested in attending, I recommend checking out their site. I'll be presenting once again about my work on visual language grammar.

Then, I'm almost more excited to say, that I'll be teaching my stuff to the Syntactic Theory class that I'm a Teacher's Assistant for here at Tufts since my professor is going to be out of town that day. My advisor is the teacher of the class, and since he pretty much helped invent modern syntactic and semantic theory, its been a thrill just being in the class let alone getting to teach a lecture or being the TA.

The course so far has aimed less at teaching the students how to do a particular theory of syntax (though it has done that a bit, advocating my advisor's new theory of Simpler Syntax), but instead at teaching them how to be syntacticians. What are the choices to be made? How can you tell what theory is best and why? These are the questions I'm struggling with in my own work, so it's been enlightening for me greatly as well. (Beyond this, I suppose my contribution just goes to make this course even more unique and weirder than the average university syntax class.)

In this spirit, I'm thinking that I'll teach the class my basic theory of VL grammar, then just give them a whole bunch of the more wacky and interesting sequences I've found and see what they can do with them. And, since I enjoyed it so much, I'm going to include Tim Godek's strip from today. Can you figure out what about it's structure makes me like it so much?

Thoughts on a visual e/i-language

Big breath; prepare for a long post...

Most of the time, people think about languages as being "out there in the world" – consisting of a conventional list of words and rules that we all access. This leads to people commenting frequently things like "that's not a word," out of the belief that there is some external authority that dictates what is or isn't in a language.

In contrast, a cognitive approach to language looks at it as a phenomenon of individual's minds, and that the mutual conventionality of many minds creates the artiface of a system that is "out there" abstracted from those individuals. Several linguists have tried to delineate the relationship of a language to its speakers.

The first major one would have to be Ferdinand de Saussure, who made the breakdown between langue and parole. While parole describes a language in use as a dynamic social activity, langue was the notion of an abstract system of expression beyond its usage.

Noam Chomsky did Saussure one better though. Chomsky maintains that this system of expression is located squarely within a human mind/brain, what he calls an internal or i-language. In contrast, and external or e-language covers both of Saussure's terms: 1) external speech behavior and 2) the system as something in the world, abstracted as we call English or Chinese.

In reality, E-langauges are built out of the mutual intelligibility of people's i-languages, which often differ based on geography and community. A good example is an irregular derviational rule like the plural of "cactus." For a Southern Californian like me, the plural is the irregular "cacti," because it's common enough in daily life to be out of the ordinary. But for, say, a Bostonian who lives around me now and never interacts with them, the prickly stuff might be called "cactuses" using the plural "s" rule of English. Both are "right" in a cognitive sense, because the cognitive structure differs based on ecological context (most jargon is like this too).

An "e-language" distinction dislikes this, yet it's the reason that dialects exist at all. They are just degrees to which people's mutual intelligiblity of i-languages group in a graded way. The "r"-less Bostonians down the road from me certainly have different rules that they follow in their i-language than I do, but our systems are close enough that we understand each other.

And... all this is setting up some useful concepts for what I really want to talk about: drawing.

I've discussed previously that there is a cultural force of an "Art perspective" that affects the development of people's drawings skills in our culture. While they may be cognitively inclined towards imitation, the Art perspective guides them towards having an individualistic style and away from using the shared structures of a community.

OR, in Chomsky's terms... the Art perspective pushes people to have i-languages that (for the most part) don't build into an e-language. There are some exceptions to this of course, maily in conventional symbols like word balloons and speed lines.

Note the semiotic allowance here though: the symbolic aspects must be conventionalized and thus easily build to a shared structure that becomes more "out there" in the world abstracted from users. But, the broader iconicity of the system (that the signs resemble what they look like) allows for a degree of mutual intelligibility without mandated conventionality. This lets individuals' styles be inexplicably tied to their own i-languages.

We immediately recognize people's styles as belonging just to them, expressing their i-language, not tied to a broader visual e-language. And, people tend to get in a huff when other people "encroach" upon one's visual i-langauge. This is the taboo of copying another person's style, instead of being looked at as building or accessing a broader visual e-language.

A result of this is the belief that images have no explicit system behind them at all, since no e-language is built. Again, since drawings are iconic, and everyone's i-languages tend to differ so much, there is an appeal to perceptual knowledge as why we can understand images, instead of a specific cognitive system at work in graphic creation.

LSA and Reductos

The last week has been utterly crazy driving up and down California visiting friends. I spent last weekend at the Linguistics Society of America conference, where I randomly ran into my TA from the first linguistics class I ever took back seven years ago at Berkeley. One of the sessions spurned a fairly good brainstorm for my visual grammar project. Now that I'm in a psychology program I was left thinking only "now how can I do this in an experiment?" I think I have a good idea for how, but we'll see what my profs think when I get back to Boston.

In other news, Grant Thomas has taken the "Reducto" graphic poem type proposed in my Comixpedia article and is applying it in a series of poems. It's fun to see someone playing with the ideas actively, so go check it out!

Dunkin' pictograms

Speaking of pictographic writing systems that will never be universal, Fabricari sends along this logo from Dunkin' Donuts:



I've been seeing that logo for over a year now, and never quite got that the images were supposed to be a pictographic sentence until a few weeks ago! I had just thought that it was a bunch of pretty pictures. Of course, given that I rarely if ever go to Dunkin' Donuts, I never really put in the effort to try and decode it.

Though, while we're here, I might as well use it as an example as to why pictographs fail as universal systems. Outright, as I mentioned in my last post, the grammar here completely mimics that of English. Of course, that was the intent in this case since its a slogan, which is why there is four units for four words. But, notice also that this conversion makes a very important decision: it chooses to transcribe "America runs on DD" as a verbal-->visual mapping, rather than siphoning the concept behind the idea into the visual modality to then adapt to its own traits. Moving on...

Let's start with nouns. First off, the DD is only understandable if you know the association to the company. The map of America could be read as "map" or some such, but is even more interesting since it is a metonymy. It uses "America" to mean "the people in America."

The verb "run" nicely shows how you can't visually show an action without also showing an object. It's tough to show "run" without also showing the "runner." Verbal grammar (by virtue of its symbolic nature) likes to divide these pieces into [ACTOR]-[ACTION]. In visual grammar this division doesn't work as well (being iconic, not symbolic), instead becoming [ACTOR:state1]-[ACTOR:state2], where "state2" shows the fruition of the action.

Finally, the preposition "on" isn't visually converted at all. I find this to be particularly telling, since it immediately shows the English context. I imagine also that the makers of the logo struggled with it, since this usage of "on" is not the spatial kind ("on top of") but is tied to the verb.

In fact, the interpretation of "run" as an action here (like "run down the street") is wholly off, since they don't mean that Americans "use their legs to run on top of Dunkin' Donuts." Rather, they are using a construction "run on" (arguably not two units) that means roughly "to be powered by." The "person running" image then becomes a "double rebus" --> first mapping the sound pattern to the image, then the image's literal meaning to it's "metaphorical" meaning.

To come back around to my initial statement about mimicing English grammar, this actually can't even do that since the slogan doesn't use concrete elements. A literal reading of this ends up being totally bizarre (bracketed by panel, italics adding clarifying info):

[The country of AMERICA][uses its legs to RUN][ON top of][DUNKIN' DONUTS]

SO... this example only goes to reinforce how hard it is to "accurately" map verbal expressions to visual signs — both individual signs and grammatical sequences — especially when it involves metonymic and metaphorical expressions (add those semantic aspects to the list then). While English speakers might be able to figure this one out, can anyone possibly imagine this working on a global scale?

Sorry, there will never be a universal writing system

Jochen Gos emailed me awhile back with news of her new book Icon-Typing and website Icon-language which is a font that creates pictograms when a person types in their own language's sentences. Unfortunately, it's taken me a rather long time to finish this post about it.

While it's a fun idea, and reminiscent of other projects of the same nature, I should say that I do not consider any of these approaches to be "real" visual languages. And, perhaps dissapointing to some, I don't consider any enterprise that strives for a "universal pictographic writing system" to be a fruitful endeavor for multiple reasons.

I do think that the idea of a pictographic language stems from a deeply felt intuition that there is indeed a visual modality language, but cannot reconcile it with the assumptions our culture makes about graphic signs and language. Among the problems holding back these types of endeavors:

1. Semiotics - There is a general misunderstanding, both in and outside of academia, that there are only two types of graphic signs: those that map to "ideas" – mistakenly like Chinese or pictures – (ideographs) and those that map to sounds - like alphabets (phonographs). This is a gross over generalization that I attacked in my thesis ¡Eye græfIk Semiosis!, trying to show that there is a much more graded system of expression at work here.

2. Grammar - All "univeral" pictographic languages must piggyback on the grammar of existing verbal languages. Verbal languages are vastly different in their grammars so much so that no possible universality could ever be reached. There is a deeper reason for this though: dividing pictograms into linearized individuated units denies the grammatical qualities inherent to the actual visual form. A grammar must arise up out of the signs themselves, not be imposed on signs from the outside (see again my paper ¡Eye græfIk Semiosis!). An Example: A pictographic system might use a marker to show plurality paired with an icon. So, a pictographic symbol might express "men" like [MAN PICTURE]+[PLURAL MARKER] (not necessarily linearly)as opposed to just showing multiple men couched in an environmental setting. In one, the concept arises directly from the sign, in the other it's latching onto the way that verbal languages do it.

3. Morphology - Believing that pictograms from all languages can easily substitute in for the words of a language presupposes that all languages have "word" units like English. What would this do for a polysynthetic language like Greenlandic or Dene that chunks multiple meaningful units into a whole syntactic unit (i.e. a single "verb" in these languages might have the same quantity of information as an English sentence). Chunking up visual pieces like this betrays a bias for how European languages break up units, not all languages.

4. Linguistic relativity - To believe a universal writing system is possible assumes that all languages structure concepts the same way. This is simply not true. The aforementioned Dene has several forms of the same verb meaning, each one reflecting a different type of object (for instance, the verb "give" changes based on the texture and character of what is being given – whether its animate, granular, squishy, flat, etc). Less "exotic" differences are like that Japanese lacks plurals and definitive articles. Plus, given how widely languages differ, which morphemes would be worthy of visual conversion? Do you convert all of them (including things like transitivity or definiteness?) or just those most "visually relevant"?... and how would that not be a subjective decision based on the preferences of the speaker of a particular language? A speaker of English might be more inclined to need particular features (like plural) that differ from a Japanese person (like politeness). A universal writing system assumes that concepts are universal to express and do so in similar ways: a big assumption.

5. Transparency and Iconic Bias - There is a general belief that Iconic signs that look like what they represent are wholly transparent in their meaning and non-culturally relative in and of themselves. This is also untrue. While most all cultures seem to be able to decode what "realistic" drawings represent, they often are subject to entrenched variability. For instance, Australian drawing systems usually feature aerial viewpoints, which can lead to odd interpretations of lateral viewpoint representations. Furthermore, simplfying the "concept bundle" of images down to a one-to-one concept-to-sign ratio is extremely difficult.

Finally, the biggest issue here is that the visual form is implicitly considered as lesser than the verbal. It uses the verbal form to be translated into the visual without stopping to think that the visual form might already have its own visual language system naturally that doesn't need a verbal connection at all. Writing systems themselves are not a bad thing, but as an importation of one expressive modality into another they will always be subject to the constraints of the imported system.

A real, natural Visual Language should be able to stand alone without needing such connections to lean on, and would reflect the diversity of different cultures as well.

License plate linguistics

Here's a little fun with writing and ambiguity. A few nights ago I saw a license plate with some friends that read:

OYABABY

Oddly enough, I parsed this first as being "Oya Baby" — drawing from my Japanese knowledge. In Japanese "Oya" is "parent", and there is a dish served with chicken and eggs over rice called "oyako don": Parent (oya) and child (ko). So, my first thought was that this plate was a Japanese oriented person making some sort of strange reference like that.

Then I hit upon another interpretation... Maybe this person was Jewish, with a nice "Oy! A Baby!"

Of course, right as I said this aloud, my friend proclaimed what seemed like it should have been obvious: "O Ya Baby!"

It seems my knowledge of Japanese and Jewish exclamations trumps my coolness.

What I particularly liked about this exchange was that each of the three parsings covered all the possible combinations for those letters:

OYA
OY__A
O__YA

And, with each parsing, the polysemous meaning of "Baby" took on a different meaning. For the first two, it was an infant, while for the second it was a familiar term of endearment (for lack of a better descriptor). (Though, you could flip the Baby meanings too: "O Ya Infant!"). It's a nice demonstration of how ambiguous written representations can be without an express phonemic link.

Of course, that license plate still isn't even close to my favorite plate from Berkeley: GRRARGH

Bizarro Scholarship

While listening today to the last lecture of this semester's Intro to Linguistics class that I TAed for, I was thinking a bit about how some of the underlying issues of scholarship in visual language contrast the way they've been approached in linguistics. By and large, I would say that my intentions for a cognitive and mind based approach to drawing and sequential images fit right in to the intents of modern linguistics. But, some of the minor issues are different.

For instance, there is often a distinction made between "Big L" Language and "little l" language. "Big L" Language is the abstract system of communication that is shared by all humans, composed of certain universal structural principles (things like syntax, phonology, semantics, morphology, etc.). "Little l" language is the instantiation of that system in the world into diverse and varied forms: English, Nicaraguan Sign Language, Chinese, Tagalog, etc. These are the variations that come from the manifestation of an abstract system.

Realizing that all these (little l) languages share a broader structure is a significant step in the study of Language. For one, it recognizes that there is a broader field that unifies the study of all the little ones.

The opposite concern occurs with regard to my notion of visual language. People already recognize a broader capacity for people to draw. It's common, normal, and uncontested as a cross-cultural phenomenon shared by all humans. However, what isnt' acknowledged is that there are "little d" drawings — that cultural forms might actually differ from each other in significant and categorizable ways.

Interestingly enough, this reverse concern yields the same ultimate result: the acknowledgement of an underlying system that is both generalizable and diverse. We can't identify individual visual languages from different cultures (say, the difference between a "superhero dialect" of American VL versus a "shojo dialect" of Japanese VL) without first acknowledging that there is 1) an underlying system behind graphic creation and 2) that the underlying system can differ between cultural populations. No matter which way you start with, the path lead to a recognition of both elements.

Other bizarro turns have been less broad. For instance, when Noam Chomsky was first arguing for generative grammar he emphasized that sentences could run to infinite lengths using finite means (ex. Phil said that Krysta said that Gina said that...). His approach sought to find a system that could have infinite expressions using finite means.

Visual sequences faced the exact opposite issue. McCloud's transitional approach to sequences of "comic" panels left no endpoint except for arbitrary physical restrictions like the end of a page or book — which have no qualitative structural impact on the system of images themselves (i.e. you could take those same images and put them into another context and they'd still process roughly the same... like on a webpage instead of a book). Transitions were inherently infinite.

When I started in on my own generative approach to panel sequences, I very quickly realized that there had to be some end for strings. So, I had to try to establish a notion of visual language "sentence." I had to argue for limitations rather than infiniteness.

I'll be on the lookout for any more of these type of contrasts...

Neil's Research: Year One Projected

Over the past few weeks my advisors and I have been planning out what my first year project is going to be. We've decided it would be easier for me to jump into a straight-up psycholinguistics study than deal with designing a "comic" based study right away. So, my project is going to be looking at ERPs and semantic coercion.

What, you ask, are ERPs? ERP stands for "Event Related Potential" and its a measurement of the electrical activity of the human brain. How it works, is the experimenter puts a cap on people that measures their EEG, or electroencephalogram, which is the ongoing electrical activity in the brain. This electrical activity is all abuzz in your noggin' all the time. So, while doing certain tasks, this cap measures these brainwaves, which are then time-locked to those events that coincide with the tasks. The waveforms are then averaged out to reduce "noise," resulting (hopefully) in a waveform that can be informative about whatever task was performed.

Or, at least, that's as far as I know so far. I technically start learning how to run subjects in our lab on Monday.

And so, I'm sure you're wondering, what is "semantic coercion"? Glad you asked... Semantic Coercion is a linguistic phenomena that occurs when certain lexical items are paired up to create a meaning that is not explicit in the sentence. For example:

Tymmi began a comic.

In this sentence, you understand that the action is beginning to read or write a comic, yet nothing in the sentence is provided that tells you that. The information is illicited out of the combination of the words. This effect is not found if that information is provided:

Alexander finished his coffee (with coercion)
Alexander finished drinking his coffee (without coercsion)

So, my question for the next year will be... is anything special going on in the brain during sentences like these?

Which brings us to a final question you might ask... does this have anything to do with comics? And the answer to that, I'm afraid, lies in a future post sometime soon...

Linguistic mythbusting

Here's an article about some of the most pervasive myths about language and questioning why linguistics is not a more prevalently studied or respected field. I personally have an answer to this: since everyone speaks language, people all feel like experts, and they think that they understand most all there is to know about it already. The same is true about graphic creation too. I have a growing list of my own "Visual language myths" that I'll post about one of these days, though I often touch on them in passing anyways.

Looking Ahead to the Past

Akin to the spirit of my last post, in the latest issue of Cognitive Science is an article about Aymara Andean metaphors for time. The authors claim that Andeans have a conception of the future as something behind the speaker and the past as in front. This is the opposite of English where most of the time we "face the future" or "look back to the past."

I have not read the piece yet, but apparently, press releases have been claiming that this reversal is unique to Andean, which the Language Log dispels quite quickly. I find it interesting that much of the data relied on gestures as well as spoken utterances, as it acknowledges the inseparability of conceptual expression in different modalities. What I'm curious about regarding this is what types of graphic narrative system this culture has (if any… I'd be floored if a culture didn't have a graphic system, but narrative — i.e. VL — is another story).

While not reliant on a front/future type metaphor, the linearity of panels in comics does echo this spatial-temporal metaphor. Would Andean allow for a similar native graphic system? I have no doubt that Andeans could learn and acquire a paneled system, but I'm more curious what sort of native system might arise, since other graphic cultural examples imply conceptual integration with language and gestures.

Hopefully before long "graphic" will become a category as centrally inquired about for studying concepts as language and gesture.

Update (6/17/06): More from the Language Log, this time a defense about the uniquness of Ayamara back/future time metaphors.

Superheroes are not Mythology

This was originally to be a blog post, then an article, and now its back to a (rather long) blog post. I hope it stimulates some good conversation. Enjoy!

I'd like to address a common thread among comic analysis/scholarship: the belief that superheroes are modern myths. While I usually refrain from discussing "interpretive" issues like this, I can confidently say that superheroes are not modern myths in any real sense comparable to the cultural functions that myths serve.

First off, myths provide an understanding of the world for people. They can be spiritually oriented, and can give insight to daily living. This is true as much for the myths followed by people practicing the dominating religions today as it was for ancient civilizations.

Often times, people think of myths as something in contrast to the belief systems we currently have, forgetting that myths are just as much a part of modern life as they ever were. At present, we have a variety of myths that have been popular for several millennia, featuring such memorable cast members as Moses, Jesus, Mohammed, Buddha, Krishna, Laozi, and many other figures. These myths inform and instruct their followers (and non-followers sometimes) on how to live good lives through the stories they tell.

Granted, superheroes might inform people's lives with moralistic advice, such as Spider-Man's "With great power comes great responsibility." However, the ethics they impart are not unique to the superhero genre, and don't do so any more than other forms of literature.

The second reason that this belief is troublesome relates back to my ever pervasive interest in language. Beyond a system of beliefs, myths also provided much more for many ancient cultures, where the stories began as oral traditions and only later became written down. For these cultures, myths created a memorization system to record and pass on knowledge.

In today's literate societies, when we want to know information, we can reference a book or the Internet. In a literate society, recording of events can be done with writing, so it can be looked up at a later date. Oral cultures lack this sort of permanent and fixed record, and in its stead myths can fill the same roles.

For example, some plants are poisonous. In our society, we can record which ones are dangerous in writing to reference and pass that information on to other people. Instead, an oral tradition might use a story of some god or spirit becoming that plant -- with some aspect of the story giving the reason for why the plant is shaped as it is.

Let me make up a myth to illustrate this:

A particularly stand-off-ish woman breaks the heart of a spirit because of her "poisonous" and "sharp" tongue. Out of despondency, the spirit transforms her into a plant with pointy leaves. Thus the plant is called a "heartbreaker," and is avoided at all cost.

Myths like this are found across the globe. It not only gives a name and reason that the plant is poisonous, but also offers a way to remember the plant through a purpose for its identifying features.

This is a practical function of mythology. These stories can then be passed on orally in a package that people can remember. It is far easier to remember a series of stories than to remember a catalog of encyclopedia entries.

Superheroes do none of these things.

Sure, superheroes may be a genre with fictional reflections of our culture. But saying that they are "myths" implies that the term means just "stories" of a fantastical nature. People have often emphasized how modern narratives follow the same structures as myths, like Luke Skywalker in the Hero role popularized by Joseph Campbell. However, this only means that these modern stories draw on the same "raw materials" as myths (or the myths themselves). It doesn't mean that they are myths. Literature and myth differ to the extant that they affect people's lives.

Of course, most myths are just stories -- but the cultural context of their use makes the difference in what distinguishes them. In many ways, I think equating modern comic book superheroes to mythology denigrates the belief systems and cultures of people whose lives are or were infused with mythology. If, and only if, superheroes can serve an equal function in modern society can they be thought of as mythological.

Once you consider the practical roles myths can play to a cultural system, superheroes carte blanche do not fulfill any of the same sorts of functions. Nor should they need to. Superheroes can do just fine as a literary genre reflecting the culture we currently live in, without needlessly attempting to be legitimized through unsubstantiated comparison to other inappropriate contexts.

Too Many Twos

I've got a short new piece up at Comixpedia for my ongoing "Comic Theory 101" column called "Two Many Twos." This one continues my series of short pieces where I try to illuminate issues involved with the intersection of “comic theory” and linguistics, here probing into the quandary called the "Problem of 2."

Like the last one, I've just tried to pose the issue and a little demonstrative visual puzzle, hoping that people might ponder/discuss it. Really, the whole thing just grew out of the last four panels, and I thought they'd be fun to work into a full piece.

I suppose this is also the first work I've done with my "avatar" as a narrator. Rather than turn myself into a kind of character, I've instead opted to keep it casual. I'd hoped for this to downplay the "me" in favor of the emphasizing the ideas more, but unfortunately in this case the ideas are demonstrated through the "me." Doh!

Genographic Project

The National Geographic will trace your distant ancestry for you with just a swab of your cheek. If I had a 100 bucks to throw around, I'd totally do it. Supposedly, us Cohn's have a unique genetic heritage anyhow, so I'm guessing one of the answers they'd give would just be "Mesopotamia."

Also, if people like reading about science, I highly recommend this blog by Carl Zimmer. His brother Ben also seems to be a linguist, who posts to the ever-entertaining Language Log.

Musings on Time and Space

One of the papers I’m currently writing is about how “time” is understood across sequences of panels. Scott McCloud’s basic position on this is that “time = space,” so moving through space means that time is passing in graphic form. I have numerous problems with this, but more than anything it has made me question why this equation might be made.

I’ve been particularly into Benjamin Lee Whorf’s writings, who argued that the language we speak affects the way we think and perceive reality. A class on this "lingustic relativity" in college is what basically motivated me to study the relationships between linguistics and “comics” in the first place.

Whorf argues that the tense system found in most European languages is what (in part) has created our sense that time is a linear thing. Because we have a past, present, and future tense, it lines up all events in a row. He makes similar arguments to the extant that we create a sense of “space” out of our quantifiers and plural system that relates back to our understanding of time.

So, this begs the question: Will the people who speak a language without a tense system (like Hopi, as Whorf shows) have a different manner of conveying concepts graphically? Would their visual languages lack sequential events entirely in favor of something else more amenable to their spoken language? Would this make the visual langauge grammar dependent on the concepts from the spoken, or can both exist with different systems of conveying events?

This is one of the most fascinating possibilities for research I think: whether the language one speaks affects the language that one draws. In some ways, this is the question that a lot of my work is leading up to. All in good time I suppose...