Equivalences for "Language"

In claiming that the graphic form (especially in sequence) is structured as a language, it might help to parse out how to make the analogy reasonably. It's not as if no one has ever noted the similarities between the forms — in fact, it's fairly common. However, mistakes are often made in my opinion.

First, I assume that there is "Equivalence" between the different modalities, which can be summarized as "the expectation that the human mind/brain treats all modalities in an equal way, given modality specific constraints."

By this account, it essentially means that we would expect all modalities to feature the same sort of storage in memory for patterned signs of differing sizes and functions (from sound patterns of phonemes to sentence patterns of idioms) and feature ways to combine all those elements at different levels of structure (phonology through discourse). We would also expect development to be similar, with a critical learning period and drop offs after that.

However, this is not the take that most comparisons of the verbals and visual forms take. Rather, they often try to make direct superficial analogies between specific types of structures. For example, "such and such" is the equivalent of a "word" or "sentence." This is often why many want to claim that single images have "grammar" — because a single image has lots of information in it, like a "sentence" and unlike a "word" — even though composition within single images behaves nothing like a grammar. (...nor should we expect it to given the differences between sound and light!)

A similar endeavor has tried to find "minimal units" of the structure of the forms, following the school of Structuralism (most popular in American linguistics from around 1920-1960ish). However, again, just knowing minimal units doesn't tell you about the broader structure, and units larger than minimal units might also be useful and insightful. It also gives no beneficial comparison other than that "minimal units" exist in both domains.

All of this is an argument for looking beyond the superficial understandings of "language" and to look for comparisons in deeper, more fundamental aspects of structuring.

Collaborative drawing

Last weekend on public television I saw a fantastic biography about Pete Seeger, the influential folk singer and activist. Throughout, Seeger stressed his desire to sing with people, not to people — motivating music as a collaborative endeavor. This sentiment is echoed in the accessible book, This is Your Brain on Music, which points out that music as "performance" by people on a stage to other people seems to be a fairly new thing. Traditionally, music was a group activity that was not reserved for those of express "skill" and training.

Drawing is much the same way. We often make a huge break between those with or without "talent" — resigning people to the misperception that they "can't draw", when really our biological endowment ensures that we all can draw. Really what is at issue is a level of fluency, and most people just don't develop with the proper exposure or practice.

Language, like this sense of music, is entirely collaborative. And, it is learned collaboratively, unlike most learning of drawing. In some cases, drawing might be instructed, often very well, though this is far from simply being interactive in the sense that you learn just by participatory immersion.

On a productive sense, drawing also is highly non-collaborative in our modern life. Belonging to a print-culture, most drawers and readers are separated by huge distances of space and time. This isn't always the case though. Sand narratives by native communities in Australia are highly interactive, drawn in real-time communication.

Humans are an intensely social animal, and my gut tells me that nearly all of our expressive capacities developed and thrive in such collaborative interactions. The question is: how in our modern ecology can we facilitate such usage for visual language? Will we have to rely on technological breakthroughs (ex. digital whiteboards), or can it grow organically without the crutch of engineering?


For those interested in more about this, my article from a few years ago "Interactive Comics" probed a lot of these ideas.

Art, Language, and "Cognitive Equivalence"

When I usually speak about the Art versus Language Perspectives, I usually couch it in a view that there are "different potential ways our society treats graphic images." As I just realized, stating it in this way maybe obscures the true intent of the distinctions.

Really, this is a hypothesis about cognition.

At the heart of my theory of visual language is the observation that we have three modalities by which we can convey our concepts: Sound, Body motions (faces/hands), and creating graphic images. That's it. To push it further, the theory is that when these modalities take on structured sequences governed by a system of constraints (a grammar), it becomes a type of language: speech, sign language, or visual language.

The "Language Perspective" assumes that all systems of conceptual expression work in similar ways — what we can maybe call "cognitive modality equivalence" or some such. Under such a view, we would expect for the graphic modality (drawings) to operate under the same principles as the verbal and manual domains.

If you look at the ways that speech and gestures (and sign language) grow developmentally in children and are used and treated in society, you see certain patterns — conventionality, imitation, communality, etc. While many of those patterns do emerge in the graphic domain, they appear "dampened", are dismissed, or just aren't recognized as such.

So, the question becomes raised: "Why don't you see these things fully in the graphic modality?" and/or "why don't we know them when we see them?"

The offered explanation is the Art Perspective — a cultural force that suppresses the patterns that would normally emerge from any other modality of conceptual expression. With polar opposite emphases, the Art Perspective works to dampen the "usual" course of development and treatment for graphic images.

So, given this, a new set of questions can be asked about this underlying "cognitive modality equivalence": What are the trends that a conceptually expressive system shows in development and society? How do modalities differ? Do these trends reflect broader cognitive processes than just conceptually expressing systems?