Over the last six months I have been thinking a lot about virtual worlds and visual, intuitive, human oriented interfaces in general. I am frankly fascinated by our increasing ability to deal with all kinds of applications in a highly visual, interactive way. After all, that is how we generally deal with the world around us. And I suspect that is how we usually think about subjects of all kinds, including those involving people.
For a while now, we have been able to deal visually with IT applications that have a physical representation, be they in science - research into molecules, pharmaceuticals and galaxies; in engineering - the design of airplanes, bridges and microprocessors; or in personal productivity - interacting with web pages, documents and e-mails; and so on. But now, because of advances in our ability to simulate and visualize just about anything, we are able to bring these capabilities to a whole new range of applications, from video games to learning; from business process designs to meetings, commerce and many other people-oriented activities.
This raises many intriguing questions in my mind. Are we better able to deal with large amounts of information visually than in text or verbal form? Is there a kind of "broadband" visual channel into our brain, vs. the perhaps more "narrowband" text and verbal channels? How many words is a picture really worth? And, most important, what are the implications of the answers to these questions for human endeavors like learning, working and shopping? Let me offer some speculations on these questions.
I read a lot, mostly work-oriented materials like e-mails, and technology and business documents, as well as general information articles in newspapers, magazines and the Web. I don't read much fiction at this time in my life, but I love to watch films. One of the appeals of films to me is that their length is usually around two hours. Given my utter inability to keep up with all the work and general interest material I'd like to read, I find it very satisfying to be able to enjoy a film in a couple of hours or so, as opposed to the many hours it would take me to read a novel.
I realize how different films and novels are, the latter usually being able to deal with characters and stories in significantly more depth than the former. But I still wonder if there is something about the visual and multimedia nature of films that permits them to tell a story in a couple of hours that would take significantly longer to read. Could it be that one of the reasons for the relative compactness of films is the fact that they are reaching our brains through a variety of channels, including the broader visual ones?
I have been trying to find out, so far unsuccessfully, the relative sizes of the portions of the brain that deal with visual images versus words. I honestly thought that this was a well understood area of brain anatomy, but if so, the answers are not easy to find. A colleague at IBM - who happens to be an IBM Fellow - thought that the portion of the brain devoted to visual activities was about two orders of magnitude larger than that devoted to textual and verbal understanding. We did some research on the Web and came up with this interesting article by Alan Blackwell from the University of Cambridge with the tongue-in-cheek title Correction: A Picture is Worth 84.1 Words.
I have contacted colleagues in the two universities I am most closely affiliated with, MIT and the University of Chicago, both of which are world leaders in brain and cognitive sciences research. While I have not yet gotten to the direct experts in either institution, the colleagues I have reached have warned me what a tricky question I am in fact asking. There is no clear-cut separation between visual and language skills. It is very difficult to estimate the size of the regions of the brain devoted to various tasks. Moreover, size by itself may not be enough, as we cannot assume that the speed of computations will be the same in different parts of the brain. I did learn that there are one million fibers in the optic nerve and 30,000 fibers in the auditory nerve. I am not sure how that relates to my original question, but perhaps it indicates that there is something to the estimate that the brain can process visual information about two orders of magnitude faster than language.
In the last few years, many in the scientific community have been wondering how capabilities coming from advanced video games can be applied to learning and training. For example, a recent Summit on Educational Games sponsored by the Federation of American Scientists observed that: “The success of complex video games demonstrates games can teach higher order thinking skills such as strategic thinking, interpretative analysis, problem solving, plan formulation and execution, and adaptation to rapid change. These are the skills U.S. employers increasingly seek in workers and new workforce entrants. These are the skills more Americans must have to compete with lower cost knowledge workers in other nations.”
Let me comment on another very important learning area where visual content and tools could have a major impact. We know that children with pervasive development disorders like autism, as well as children with milder learning disabilities often have trouble reading and processing verbal commands.
One approach now widely used to teach language to autistic children is the use of highly repetitive drills involving pictures, speech and text. I know this from personal experience. Such drills were the only way that my own son – who was diagnosed with autism at an early age – was able to learn to understand and use language. Perhaps what works with this approach is the use of a variety of channels into the brain, including the visual ones, which is often not impaired to the same degree as the language channels in children suffering from pervasive developmental and learning disabilities.
In addition, I have been learning about recent brain research into language development, and in particular, the importance of stimulating the brain of very young children by reading and talking to them and exposing them to a large vocabulary. Otherwise, language development can be seriously impaired. Once children enter school, they will be mostly taught via language - verbal and reading. So those children whose language processing in the brain did not properly develop due to lack of proper stimulation will be at a very serious disadvantage. It is not surprising that children who grow up in underprivileged environments, whose families are often unable to provide them with the proper nurturing and stimulation, will have serious trouble learning once they get in school, fall further and further behind and eventually fail.
Thus, as technologies enable us to present information in a more visual and human oriented way, there are many serious applications and problems in society that we can now begin to address. This will require much research to realize, but It is no wonder that so many of us are so excited about the possibilities.