Over the last six months I have been thinking a lot about virtual worlds and visual, intuitive, human oriented interfaces in general. I am frankly fascinated by our increasing ability to deal with all kinds of applications in a highly visual, interactive way. After all, that is how we generally deal with the world around us. And I suspect that is how we usually think about subjects of all kinds, including those involving people.
For a while now, we have been able to deal visually with IT applications that have a physical representation, be they in science - research into molecules, pharmaceuticals and galaxies; in engineering - the design of airplanes, bridges and microprocessors; or in personal productivity - interacting with web pages, documents and e-mails; and so on. But now, because of advances in our ability to simulate and visualize just about anything, we are able to bring these capabilities to a whole new range of applications, from video games to learning; from business process designs to meetings, commerce and many other people-oriented activities.
This raises many intriguing questions in my mind. Are we better able to deal with large amounts of information visually than in text or verbal form? Is there a kind of "broadband" visual channel into our brain, vs. the perhaps more "narrowband" text and verbal channels? How many words is a picture really worth? And, most important, what are the implications of the answers to these questions for human endeavors like learning, working and shopping? Let me offer some speculations on these questions.
I read a lot, mostly work-oriented materials like e-mails, and technology and business documents, as well as general information articles in newspapers, magazines and the Web. I don't read much fiction at this time in my life, but I love to watch films. One of the appeals of films to me is that their length is usually around two hours. Given my utter inability to keep up with all the work and general interest material I'd like to read, I find it very satisfying to be able to enjoy a film in a couple of hours or so, as opposed to the many hours it would take me to read a novel.
I realize how different films and novels are, the latter usually being able to deal with characters and stories in significantly more depth than the former. But I still wonder if there is something about the visual and multimedia nature of films that permits them to tell a story in a couple of hours that would take significantly longer to read. Could it be that one of the reasons for the relative compactness of films is the fact that they are reaching our brains through a variety of channels, including the broader visual ones?
I have been trying to find out, so far unsuccessfully, the relative sizes of the portions of the brain that deal with visual images versus words. I honestly thought that this was a well understood area of brain anatomy, but if so, the answers are not easy to find. A colleague at IBM - who happens to be an IBM Fellow - thought that the portion of the brain devoted to visual activities was about two orders of magnitude larger than that devoted to textual and verbal understanding. We did some research on the Web and came up with this interesting article by Alan Blackwell from the University of Cambridge with the tongue-in-cheek title Correction: A Picture is Worth 84.1 Words.
I have contacted colleagues in the two universities I am most closely affiliated with, MIT and the University of Chicago, both of which are world leaders in brain and cognitive sciences research. While I have not yet gotten to the direct experts in either institution, the colleagues I have reached have warned me what a tricky question I am in fact asking. There is no clear-cut separation between visual and language skills. It is very difficult to estimate the size of the regions of the brain devoted to various tasks. Moreover, size by itself may not be enough, as we cannot assume that the speed of computations will be the same in different parts of the brain. I did learn that there are one million fibers in the optic nerve and 30,000 fibers in the auditory nerve. I am not sure how that relates to my original question, but perhaps it indicates that there is something to the estimate that the brain can process visual information about two orders of magnitude faster than language.
In the last few years, many in the scientific community have been wondering how capabilities coming from advanced video games can be applied to learning and training. For example, a recent Summit on Educational Games sponsored by the Federation of American Scientists observed that: “The success of complex video games demonstrates games can teach higher order thinking skills such as strategic thinking, interpretative analysis, problem solving, plan formulation and execution, and adaptation to rapid change. These are the skills U.S. employers increasingly seek in workers and new workforce entrants. These are the skills more Americans must have to compete with lower cost knowledge workers in other nations.”
Let me comment on another very important learning area where visual content and tools could have a major impact. We know that children with pervasive development disorders like autism, as well as children with milder learning disabilities often have trouble reading and processing verbal commands.
One approach now widely used to teach language to autistic children is the use of highly repetitive drills involving pictures, speech and text. I know this from personal experience. Such drills were the only way that my own son – who was diagnosed with autism at an early age – was able to learn to understand and use language. Perhaps what works with this approach is the use of a variety of channels into the brain, including the visual ones, which is often not impaired to the same degree as the language channels in children suffering from pervasive developmental and learning disabilities.
In addition, I have been learning about recent brain research into language development, and in particular, the importance of stimulating the brain of very young children by reading and talking to them and exposing them to a large vocabulary. Otherwise, language development can be seriously impaired. Once children enter school, they will be mostly taught via language - verbal and reading. So those children whose language processing in the brain did not properly develop due to lack of proper stimulation will be at a very serious disadvantage. It is not surprising that children who grow up in underprivileged environments, whose families are often unable to provide them with the proper nurturing and stimulation, will have serious trouble learning once they get in school, fall further and further behind and eventually fail.
Thus, as technologies enable us to present information in a more visual and human oriented way, there are many serious applications and problems in society that we can now begin to address. This will require much research to realize, but It is no wonder that so many of us are so excited about the possibilities.
=====
Could it be that one of the reasons for the relative compactness of films is the fact that they are reaching our brains through a variety of channels, including the broader visual ones?
=====
That would be my assumption. Its certainly the case in communication. There is much more effective information sharing in richer forms of communication (e.g. face to face) than there are in less rich forms (e.g. IM). I'm not familiar with the neuroscience, but I certainly can speak from experience in saying that I can consume a movie scene much faster than the same scene in a book since the emotion / mood / movements of the characters are sent in parallel data channel.
Posted by: Frank Jania | February 18, 2007 at 11:58 PM
my visual processing centers are pathetic. i am almost all text. you might take a different tack and use an online poll to find out whether people are visually or language oriented. i am sure the geniuses at MIT have some useful data too, but it probably wouldn't hurt to ask people how their minds work. i dont see films as particularly compact. how many times has someone said - oh you need to see it a few times to get this or that (shot, gag or whatever) - with books people rarely say oh you need to read it a bunch of times to get the value. that could be because movies are less compact in that they are designed to wash over us, rather than asking for in process feedback. the two hour movie meanwhile is very often just a five piece plot extended into set-piece heaven. i
Posted by: james governor | February 19, 2007 at 08:36 AM
Irving, I remember when you were discussing this two weeks ago in RTP. You made a particularly humorous remark that I thought was right on, about the bits 'falling out of your head' sometimes with the information overload that we deal with. I truly hope that some of the brain research underway today starts looking into these questions about the possibility of having different learning channels other than typical language processing. I constantly feel that there are so many things that I want to read and process going on around me, that it does become very hard at times to hang on to the information. Thank you for the insight!
Posted by: Manuel Monserrate | February 19, 2007 at 09:27 AM
Irving, I remember when you were discussing this two weeks ago in RTP. You made a particularly humorous remark that I thought was right on, about the bits 'falling out of your head' sometimes with the information overload that we deal with. I truly hope that some of the brain research underway today starts looking into these questions about the possibility of having different learning channels other than typical language processing. I constantly feel that there are so many things that I want to read and process going on around me, that it does become very hard at times to hang on to the information. Thank you for the insight!
Posted by: Manuel Monserrate | February 19, 2007 at 09:27 AM
Irving, your question "if there is something about the visual and multimedia nature of films that permits them to tell a story in a couple of hours that would take significantly longer to read." ist very interesting and you should try a little experiment.
Take two novels that have been made into films, one with a real world setting and one with something you are not used to have a visual impression from (as for example a Science Fiction setting).
I think you will agree that you could sort this 4 different media impressions as follows (least time needed for reception on top):
film in real world setting
film in sci-fi setting
text in real world setting
text in sch-fi setting
And my guess is, that if you are seeing a familiar shape or hearing a familiar sound your brain takes a shortcut in digesting the se information.
I sometimes do some training for people (Information Workers) to help them in optimizing their work and one of the techniques (in Mind Manager Mapping) I show them is using color schemes and symbols they are accustomed to and that are commonly agreed upon so that they and other people they give their maps to understand the meaning very fast.
Best wishes
Andreas
Posted by: Andreas Weinberger | February 19, 2007 at 10:47 AM
aha. see this take on infant brain cognition- and why fast-moving images are not so good.
"# Talk to young children. Because language is symbolic, it requires thinking just to "unlock" meanings of words.
# Use audio tapes frequently to replace videos or movies for entertainment. Fast-moving screen images require little thought to process and don't give much time to the cerebral cortex to engage.
# Limit the use of television. TV is not very symbolic and usually requires few mental gymnastics to interpret. Remember, when the brain is used to thinking, it actually grows in mass"
http://www.nncc.org/Release/brain.language.html
Posted by: james governor | February 19, 2007 at 11:31 AM