(^_^) ... Emoticons and the Brain

Masahide Yuasa, Keiichi Saito,Naoki Mukawa. 2006. Emoticons convey emotions without cognition of faces: an fMRI study. Conference on Human Factors in Computing Systems. April 22-27, 2006. Montréal, Québec, Canada

Related to the previous post on a testing of McCloud's "Cartoon Identification Theory" on cartoony vs. realistic images in the brain, here's a study using fMRI (brain scans) to look at emoticons which at this point are perhaps the most simplified signs for faces we use.

Faces have been the focus of a lot of debate in cognitive neuroscience, particularly about the "face area" in the brain. One side says it's an area strictly devoted to processing human faces, the other side says that it's an "expertise" area and it activates because humans are experts at recognizing faces. It is one of the most fiery debates in cognitive neuroscience, and learned about in most all intro classes.

Amazingly, this study shows no activation of this "face area" when looking at emoticons. :-O

Using fMRI, the authors compare Asian style emoticons (non rotated) with averaged faces (photos of multiple faces that have been blended to be more "generic") that were expressing the emotions of happiness and sadness. Emoticons appeared first on their own, and in a second study embedded within sentences, while non-emoticon signs using the same characters were also used as fillers (i.e. ":O*-<").

They found that photos of faces activate both areas pertaining to emotional valence (right inferior frontal gyrus) and facial recognition (right fusiform gyrus), while emoticons only activate emotional areas but not face areas. That is, as the authors say, "Remarkably, emoticons convey emotions without cognition of faces."

This finding has very interesting consequences for understanding how brains process varying degrees of complexity in images. The implication here at least is that more simplified faces become tied more explicitly to a "symbolic" meaning as opposed to their iconic meaning of resembling what they look like. That is, more simplified images strip down the meaning to its core meaning disconnected to the iconic reference that they are framed within.

It would be interesting to see a graded approach to this — such as taking different degrees of representation from McCloud's gradient of "cartoonification" (or to use my term, "haplosis"). Are there different degrees of activation for different representations? Does activation for the fusiform gyrus all suddenly drop off at a certain level of simplification? How does this affect the debate that the fusiform gyrus is an "expertise" area rather than a face area?

(^_^)

Why Do You Make Comics write in visual language, If You Do?

Over at TalkAboutComics, Joey Manley asks "Why do people draw comics?", and makes the observation that...

A lot of people who read comics also make them. Maybe even most people.
...
Almost everybody I’ve ever met who reads comics has, at some point or another, actually made one, even if he or she never showed it to anybody.

To me, this has to do with the nature of the visual language of sequential images. Unfortunately, our failed definition of "comics" conflates the idea that "comics=sequential images", when really "comics=cultural context" and "sequential images=visual language." By this notion, what Joey is talking about is that comic readers are highly likely to create stories (comics) using this visual language.

Perhaps this should be unsurprising then, since it means this visual language works like any other language. People get exposure to it, and imitate it in order to learn and practice their fluency.

This is not a case of people "drawing comics" the way that filmmakers "make movies" or other types of "artistic" craftsmanship. Rather, this is more like the way that Americans "speak English" or Quebecois "parle en Français" because those are the languages of the communities to which they belong. Comic readers constitute the language group for visual language in America, so their "drawing comics" is simply participating in the (visual) language of their community.

Note that this also applies to the particular graphic dialect that they might partake in: those who read manga are likely to draw like manga, those that read superhero books are likely to draw that way.

Split planes

I haven't gotten a newspaper at home in years, but every now and then my father sends me clippings of comics or articles. The comic that he seems to send the most is Pearls Before Swine which has a periodic flair for formalism.

In the last set that he sent, the characters do a fair amount of walking on the borders of the panels (Here, Here, and Here):



"Awareness" of panel borders by characters within them is nothing new, but doing so it reveals that there are two levels of representations in this visual language of comics. There is a "Representational Plane" (RP) that the content exists in, and a "Framing Plane" (FP) that holds things like panel borders and balloons/bubbles/text boxes. Usually, the Framing Plane just lies "outside" the RP, but instances like these collapse the layers together. (see linked essay below for illustrations of this)

Another hint that these two layers exist comes from the fact that text carriers can become panels, as I discussed in my article on "Loopy Framing":



This commonality between their forms — that they both encapsulate information, both are not part of the image matter but can be interacted with in a "meta" way — go towards their being two aspects of a singular plane of Framing.

Note: For those more interested, I discuss this more extensively in my paper Interactions and Interfaces.

Double Standards in Style

I had an interesting realization the other day about the way people judge the quality of realistic versus cartoony drawing styles. It seems to me that the more someone tries to maintain a realistic style, the more harshly criticized they will be when they don't "fully achieve" it. Cartoony styles get no critique like this.

As I've done before, perhaps Rob Liefeld will be a good example. Liefeld is often criticized for his unrealistic body proportions and suspect anatomical correctness — despite being proficient in his craft. However, I've never heard of Matt Groening criticized as having a poor understanding of anatomy for the Simpson's only having four fingers, or that practically no one has chins.

It seems to me that with cartoony styles, we accept that drawings are more of a representation of a concept than a re-creation of "reality." The more realistic the style becomes, the less accepting people are of this, yet it still remains true: The capacity to draw is for representing thoughts visually.

Graphic safety for airplane crashes

One of the reasons I've been a bit MIA from the blog lately is because I've been traveling a lot. On one flight, I noticed some interesting things in the safety manual. First, let's look at a section that isn't too bad:



This part shows a nice step-by-step for what to do in the event of increased cabin pressure: how to put on an oxygen mask. Note the "no smoking" sign to the side — smoking with one of those things would be bad. Here, the numbered panels may serve two purposes — it gives a reading order along the "z-path", left-to-right and down, and it also gives an order to the procedure. You need to follow these steps in order to put the mask on correctly.

Comparatively, this one from right below seems very strange:



If the numbers are meant to direct a stage-by-stage process (or even just an order for reading the panels), then perhaps I can paraphrase the overall meaning (and the signs to the side):

"In the event of either a land or water crash, first, African Americans need to cover their heads. Then, blond women need to grab their legs, followed by white men who should force their children down next to them into submission. Lastly, once all the other groups are safe, pregnant women should brace themselves. They're the bottom rung in our concerns."

Ok, so maybe I added a few minor embellishments, but my point should be clear: this sequence has no need of numbers for any purpose. Really, all people should brace themselves as quick as possible in one of these ways, not in any particular temporal order. And, despite that people probably would read it in a z-path anyhow, there isn't any need to read these in the numbered order here either.

I love this example especially because it provides great support for why sequential images are not always a sequence in time. Each of these panels simply shows a different viewpoint of a broader scene. It's a shift in Space but not in Time. Truly, you could rearrange these panels in any order and still maintain the same meaning — clearly a sign that no "time" passes across the panel boundaries.

Ambiguous signage in words and pictures

Things have been crazy busy lately, so blogging has been a bit sparse. However, here's a few signs I found around the Boston area that struck me as interesting for their ambiguity. First off...



This sign posted outside a fence to a lumber-yard features a line I've noticed more and more around here, which is "Police Take Notice." What initially amused me about the sign, and this phrase in particular, is that this can be read in two ways.

The first, what I assume is the intended way, says something along the lines of "No Tresspassing. Police will notice if you do."

The other interpretation is directed at the police: "No Tresspassing. Hey Police, that means you! Stay out!" I just pictured some guy with no shirt, overalls, and a shotgun looking out at his lumber hoping no cops come around.

Here's the other sign, from a street post:



This one is clearly trying to prevent domestic abuse, fairly clearly stated by the text. However, the image can be a little ambiguous. Given that only two hands are shown without connection to other bodies, it is technically unclear whether they belong to the same or different people. The different readings give very different interpretations.

Again, starting with the intended view, they belong to different people. The fist of one person is being stopped by another person — the illustrated stopping of the fist of domestic violence.

However, if you perceive the two hands as belonging to the same person, it seems like the common gesture of one person grinding or punching their fist against their own open hand as if itching for a fight. This gets just the opposite meaning, since it implies a desire/threat for violence.

Manga facial symbols

I've been putting the finishing touches on my first draft of a book chapter for a textbook on Japanese manga recently. Mainly, the last things I had to do was draw my examples. Also being a "comic author" is a big advantage as a scholar, since I can create my own examples when I don't want to deal with issues of copyright and permissions.

For this one, the challenge was drawing in the "manga style," which I rarely do. For better or worse, here's some symbolic faces I've whipped up:

This section on graphic symbols/conventions was one of the hardest portions of the actual paper to write. While most of us know that manga uses a ton of wacky conventions, there aren't many places referencing them outside of informal listings like wikipedia. At most, various sources mention one or two different conventions, but I couldn't find any extensive type of cataloging. (though, if anyone is aware of such a thing, please let me know)

I started trying to make a cross-cultural list like this back when I used to have the forum, but that project seems to have stagnated. This is a research project just waiting for someone to take it up (like oh so many)...

Understanding motion lines

Friedman, Sarah L., and Marguerite B. Stevenson. 1975. Developmental Changes in the Understanding of Implied Motion in Two-dimensional Pictures. Child Development 46:773-778.

This paper reports on an experiment testing how action lines are understood by children. They compared figures with postural cues showing motion (i.e. they look like they're running) with those with motion lines, and those that have polymorphic features (i.e. repeating legs over and over to show motion). They tested preeschoolers, first graders, sixth graders, and college students.

Massive increase in the understanding of “cartoon” conventions (i.e. motion lines) occured between the ages of first grade and sixth grade. (possibly due to developmental reasons... or to increased exposure to comics?). Reliance on postural cues decreases from first grade through college. Understanding of polymorphic representation increases greatly between pre-school and first grade, then levels out. The insinuation is that conventional cues are relied on more and more as people age.


Gross, Dana, Nelson Soken, Karl S. Rosengren, Anne D. Pick, Bradford H. Pillow, and Patricia Melendez. 1991. Children’s Understanding of Action Lines and the Static Representation of Speed of Locomotion. Child Development 62:1124-1141.

A similar study also tested children’s knowledge of action lines to determine whether body posture or action lines contributed more to understanding with seven and nine year olds, and adults. The stimuli showed a running figure (one silhouetted, one photo-like), without lines, with lines trailing it, or with lines behind it. They also had a task where subjects drew their own figures running.

They found that the relationship of action lines to the meaning of locomotion is non-arbitrary, though exposure to drawings using it is necessary to understand its convention. Again, children attenuated more to postural cues than adults did. The younger children did not distinguish between action lines and background lines to the same degree as older children and adults.

Interestingly, it also asked subjects what the lines represented, showing that children contributed meaning to lines where adults did not. Many children gave visible and invisible explanations for the lines – such as “air moving” or “wind.” Adults simply accepted the lines as symbolic representations. In cartoon drawings, adults treated lines as “path-of-movement metaphors,” but for photos were treated as non-arbitrary cues for perceptual movement.

---

(Updated 8/25): These conclusions are interesting for a few reasons... one, it shows that not all graphic things are universally or transparently understood by everyone. It takes some degree of development to reach that understanding for things that we take for granted as obvious. This is different than some of the findings on speech balloons or thought bubbles.

The fact that the development happened most in pre-puberty aligns with many other developmental changes, like language acquisition. Whether that development has to do with exposure or just age, it's hard to tell.

Also, its interesting that there was a change in how they considered what the lines were — from an iconic meaning (claiming the lines are "wind") to a purely symbolic meaning. This too is consistent with developmental changes in other domains.

Perhaps I should avoid the slopes...

After the conference in New Mexico let out, a bunch of us went on an excursion to the top of a mountain that overlooks Albuqurque. At the top, we found this sign on the side of a building:



Given that we'd all just talked about visual signs for a day and a half, many of us photographed it and cracked jokes. While one might expect the images are there to help give aid to those who don't speak the language or to reinforce the text, but really most of them are extremely ambiguous...

The top left one looks like the skiier is walking on cracks in the earth, and the bottom middle one looks like it's telling you to hang out in the middle of the road (when the text says exactly the opposite). The top middle one seems like the skiier is dropping his skiis from the lift.

My favorite though, is the upper right. It could easily be construed that, when you're skiing and approaching someone else, you should go around them on both sides at the same time (except for its physical difficulties). I kept picturing some Looney Tunes cartoon, where someone hikes up their body to elongate their legs so they weave around an object to avoid hitting it.

If one can only read the very large text at top (since the smaller text is a bit tough to read), these "signs of safe skiing" paint an awfully strange picture of what skiing is like.

Anyone got any more good interpretations of these?

Thought Bubbles, Children, and Autism

These are several articles I found particularly interesting looking at the understanding of thought bubbles, and using them to help autistic children.

Wellman, H. M., Hollander, M., & Schult, C. A. (1996). Young Children’s Understanding of Thought Bubbles and Thoughts. Child Development, 67, 768-788.

Several experiments on 3 and 4 year olds show that thought bubbles are understood at fast rates by both age groups as depicting thoughts. This is interesting, because at similar ages, other visual conventions such as speed lines are acquired over age (I'll post on these articles next maybe).

This contrasts with the findings in the Yannicopoulou study, which showed that preliterate children could not distinguish speech from thought balloons – which was not tested in these experiments. They suggest that this shows evidence for “something recognizable from our everyday understanding or experience of thoughts themselves” tapping into "theory of mind" knowledge...

Wellman, H. M., Baron-Cohen, S., Caswell, R., Gomez, J. C., Swettenham, J., Toye, E., et al. (2002). Thought-bubbles help children with autism acquire an alternative to a theory of mind. Autism, 6(4), 343-363.

Children with autism have specific difficulties understanding complex mental states in other people like thought, belief, and false belief and their effects on behavior (what are known in psychology as "Theory of mind").

These children benefit from focused teaching about thoughts, where beliefs are likened to photographs-in-the-head. Here two studies, one with seven participants and one with 10, tested a picture-in-the-head strategy for dealing with thoughts and behaviour by teaching children with autism about cartoon thought bubbles as a device for representing such mental states.

This device led children with autism to pass not only false belief tests, but also related theory of mind tests. These results confirm earlier findings of the efficacy of picture-in-the-head teaching about mental states, but go further in showing that thought-bubble training more easily extends to children’s understanding of thoughts (not just behaviour) and to enhanced performance on several transfer tasks. Thought-bubbles provide a theoretically interesting as well as especially easy and effective teaching technique.

Pre-literate understanding of speech balloons

Yannicopoulou, A. (2004). Visual Aspects of Written Texts: Preschoolers View Comics. L1- Educational Studies in Language and Literature, 4, 169-181.

This study assessed preschool aged children’s ability to understand various features of Carriers (i.e. thought bubbles and speech balloons), despite not being able to understand written language. Tests showed:

- 87.1% recognized angular balloons meant anger.
- 83.7% correlated a flowery border to politeness.
- 78.7% recognized increase in volume by increase in size.
- Speech vs. thought balloons were distinguished for their meaning at chance (49.7% speech/47.5% thought)
- Most poor was recognition of other languages as indicated by variation in text.

These results were fairly interesting, especially since they imply that children can recognize aspects of manner of speaking (politeness, anger, etc), yet can't differentiate plain thoughts versus speech. Part of this might relate to a general developmental trajectory though — that children don't yet have "theory of mind," the recognition that other individuals' have thoughts of their own. Preschool children are roughly at the age where this ability is developing, so the problems they had recognizing thought bubbles might be due to their lack of understanding thoughts in other people in general.

However, these data contrast with other studies in this regard that I'll be posting sometime soon.

Blindfolded patterns

Dirk links to this interesting post where cartoonists were asked to draw their characters blindfolded (versions on the right). This Chester Gould one is pretty impressive...



What's interesting to note is where most had problems and where they were fine. Most of the linework between the blindfolded and sighted versions are the same, it's putting them all together that they had trouble with. I like this a lot since it shows that authors are using consistent patterns when drawing (as if people didn't know this), even when they can't see the page!

This all reminds me of some citation I saw that claimed blind people also use/conceive of speed lines to indicate motion, but I never fully got the chance to read up on it. Perhaps I'll hunt that one down...

Mannerism, Imitation, and Iconic Bias

Dirk (who kindly keeps linking to my posts) provides some thoughts on my last post about imitation by describing "Mannerism" (about halfway down the post).

He describes the Renaissance Mannerists (what I'll call abstractly as "cohort 2") copied the style of the first wave of Renaissance artists like Leonardo and Michaelangelo that came before them ("cohort 1"). Where cohort 1 actually studied the anatomy of real animals and people, cohort 2 simply copied cohort 1. As a result, cohort 2 is said by Dirk to have...

"the surface tics of the Rennaissance masters down pat, but his work displays none of the anatomical understanding by which they came to be able to create such accomplished illusions of form and light. Mannerism is an artistic game of Whisper, with details lost and distorted as they move further away from their point of origin."

Dirk goes on apply this same process to describe the work of Rob Liefeld and others (who I've oddly defended on this front before).

Now, this quote to me summarizes almost exactly the points I made last time that are inherent to the "Art" perspective. With Mannerism, it's not just about copying — it's that cohort 1 didn't copy at all, they drew from real life. This is what I've called Iconic Bias in past posts (parts one, two, three): The belief that the graphic modality of expression should resemble real life ("iconicity" in the semiotic sense). The "purity" of that first cohort is drawn from iconicity, and the lack of it in cohort 2 leads to their derision.

My response is that this isn't the way the human mind is primed (the "Language" perspective): the mind is primed for imitation, and any drawing "style" is a reflection of mental patterns that have become habituated within a drawer's long-term memory. Those patterns become set in this case through one of two ways: 1) copying other people's patterns, or 2) copying perception and siphoning that iconicity through one's mental structures.

The "Art" perspective says that only choice two should be acceptable, with minimal influence from choice one. Recall for instance, McCloud's Six Steps of learning from Understanding Comics: His first level is imitation, but then all subsequent steps require one to cast aside all other influences.

But, as I've pointed out in the past, rejecting the influence of any cohort before your own works against the establishment of conventional signs — which are what language is made up entirely of. The only reason there is a "graphic dialect" of a superhero style at all is because of imitation. Manga thrive on a style that was founded on imitation (Tezuka being largely considered cohort 1, but Walt Disney and others being cohort -1 for him).

Imitation hasn't hurt manga at all. In fact, I'd argue that it has probably helped them in numerous ways: 1) A consistent cultural style allows more focus to be placed on what that style is used to express story-wise than so much focus on the surface depictions. 2) A consistent style across numerous authors is more readily accessible to young readers, especially those who want imitate them. In America, when children want to "draw comics," they want to draw stories about stuff. But, when kids want to "draw manga" they want to draw stories in the style of manga because that's the visual vocabulary that they are now exposed to.

This is just like language: "Exposure + practice = fluency." With language, successive cohorts are always the manner by which it is transmitted. A great example of this is Nicaraguan Sign Language, where several deaf children who had created their own gesture systems combined their contributions to make cohort 1. Successive cohorts took what they did only to refine and alter it into further grammatical patterns. With the anti-imitation influence of Art, this process of conventionalization is largely lost (outside symbolic signs like word balloons and speed lines at least).

The Art pespective just wants to substitute the cognitive man-made exposure for that of real life, and with that, jettisoning an idea of fluency (proficiency in a system) for skill (accuracy at depicting real life): "Perception + practice = skill at representing perception."

While I won't go into it at length, I find it intriguing that in Dirk's same post, there is a damning attitude for Greg Land, who takes iconicity to the extreme by drawing wholly from photo reference — only that he picks and chooses parts of photos to combine thereby messing up the anatomy. So, here it seems to be the case of messing up iconicity through the most iconic method possible!


Final note, so my intentions aren't misunderstood: I should point out that this is not a post of advocacy; I'm not saying people should or shouldn't copy other people. I'm just trying to analyze the issues involved, and in some case, defend all strategies as being cognitively acceptable.

----
Update 2/26: Dirk has a short reply to this post (halfway down). I don't have much in response to it except that it still maintains the "Iconic bias" underlying the last couple posts . Beyond that, he makes some interesting points.

"Cartooning Symbolia"

Derik points to this comic, Cartooning Symbolia, by Dash Shaw that uses Mort Walkers terms from The Lexicon of Comicana for various symbolic elements of the graphic medium. It's an interesting experiment in formalism, and I especially appreciate the last twelve panels that introduce "new symbols," which well exemplifies the complete conventionality of symbols given that the reader has no idea what these things mean.

I read Walker's book for the first time last year, and here's what I wrote back then to the comixscholars list:

Its not a bad read. Its funny and lighthearted. One gag even got me to laugh out loud. He does point out an awful lot of graphic conventions used throughout many American comics – particularly strips – and makes a few interesting observations about them. Its by no means exhaustive, though it does have a surprising amount in it.

He also attaches a myriad of useless names to them, to the extant that you feel that his whole point for jargon is to be facetious (which it may well have been). You can tell that some of his terms have a logical origin to them, while others just seem made up because he wanted to give everything a name. I also have some discontent with his organization of these things, but for very specific structural reasons that I will bring up in some future writings of my own.

This organizational issue is related to my own attempts to compile a list of conventional signs in visual language. Many of the things he puts as separate signs I will include together in a larger category (like "smelly lines" and "sun rays" as both types of "path lines"). And while I'm at it... my list is ongoing, so please alert me to any more signs of this nature if you come across them. Perhaps someday I can compile them into more of a dictionary/wiki type project with examples and whatnot...

Styles and genres

Cross-posted to comixpedia... I think a reasonable argument can be made that certain styles insinuate certain genres. This isn't to say that genres are drawn in those styles all the time or that those genres can't use other styles, but that when we see a certain style, we think of a certain genre. I'd like people's help in formulating a list of these, if we can...

In American styles, I have:

Superhero
Cartoony
Dramatic/Romantic comic strip (?)
Art comix

This is of course leaving out imported styles like "manga"... I'd even be up for hearing "substyles" within genres too. Got any more? (examples are always helpful)

Digital Creation

As evidenced by my work in A Love Story and the Meditations series (like the now updated "Karuna" story), I have embraced digital tools with open arms. Though, I have some thoughts on their use, especially integrating them with line art and, shall we say, “techniques from the hand” (i.e. non-CGI produced).

Essentially, I think that digital tools should be embraced, but used in careful moderation. Often, digital graphics are extremely pristine, smooth, and uniform. Comparatively, “hand done” works are messy, imprecise, fallible, random. And I think that at least some level of inherent "mistakes" are important.

For instance, I generally can’t stand 3D CGI art in comics when its used dominatingly. I think it just looks wrong, mainly because it lacks a sense of randomness. It is too clean. Human beings make mistakes, and those mistakes are part of what make us human. Without those elements reflected in our “art,” the results seem cold and artificial (unless that’s the aim of course).

More so, on a level of theory, using a completely CGI creation lacks the cultural conventionality inherent in a “drawing style.” It is the epitome of striving for iconicity, though here at the expense of rooting graphic creation in cognitive structure.

Personally, given the available tools, I strive for a balance of these elements. Context dependent, I want the randomness of line art, the precision and naturalness of photography, and the clean smooth uniformity of digital graphics.

Gestures in comics

A doubleshot of reviews:

Fein, Ofer, and Kasher, Asa. 1996. How to do Things with Words and Gestures in Comics. Journal of Pragmatics, Vol. 26.6. Dec. 1996. Pp. 793-808

This study looked at the role of gestures in comics (specifically, those in the European comic Asterix). The study had people interpret the meanings of both panels from the comics, and of photos where people took on similar poses. The backgrounds of the panels were erased, so there was no context for the gestures. In one part, they were asked to write possible dialogue for the gestures, and in another task they were given potential meanings and asked to assign them.

It concludes that gestures in comics are interpreted the same as ‘real life’ gestures, and that the meaning imbued in them comes from the ingesticular force (i.e. the intent of the expression) rather than the propositional content of the accompanying speech (in word balloons). One interesting tidbit noted that some people said the photos were actually harder to interpret than the comics panels (though the stats disputed this). If this were true, then it would support McCloud's insinuation that cartoony images are more "base" than realistic ones. I'd like to know the VL fluency of the subjects and whether people with more "comics" experience rated higher or lower in this regard.

Raecke Jochen. 1999. Using Comics as Data for Research into the Connection between Pointing Gestures and Deictics. In E. André, M. Poesio, and H. Rieser (eds). Proceedings of the Workshop on Deixis, Demonstration, and Deictic Belief at ESSLLI XI.

Uses comics to analyze the relationship between deictics and gestures in Serbo-Croation. His method codes a corpus of comics comparing the relations of the images' gestures to the conent of the speech balloons. He finds that pointing gestures by far dominate the gestures, and pointing gestures alone do not fulfill the meaning of the representations (i.e. multimodality is necessary). This isn't surprising, since pointing gestures are indexical, which means that they only indicate meaing in something else (the same way a pronoun refers to a different element for meaning).