Tom Malone gave a very interesting talk on collective intelligence at the IBM Cognitive Systems Colloquium which I recently attended and wrote about. Malone is Professor of Management at MIT’s Sloan School and the founding director of the MIT Center for Collective Intelligence (CCI). His research is primarily driven by this fundamental question: “How can people and computers be connected so that - collectively - they act more intelligently than any individuals, groups, or computers have ever done before?” This is a very important question to explore to help us understand the impact of our increasingly smart machines on the very nature of work and organizations.
Malone and his collaborators are conducting research on a number of topics in this area. Do groups exhibit characteristic levels of intelligence which can be measured and used to predict the group’s performance across a wide variety of cognitive tasks? If so, can you devise tests to measure the group’s intelligence using methodologies and statistical techniques similar to those that have been applied to measure the IQs of individuals for the past hundred years?
To answer these questions they conducted a number of studies where they randomly assigned individuals to different groups, which then worked on a variety of tasks. Their research results were published in the October 2010 issue of Science.
The studies measured the individual IQs of each of the participants, and found that group intelligence was only moderately correlated with the average and maximum intelligence of the individual group members. But, they did find three group attributes that were statistically significant factors in predicting how well each group will do on a wide range of tasks. One was the average social sensitivity and perceptiveness of the group, that is, the ability of group members to read each other’s emotions. They also found that groups in which a few people dominated the conversation did not perform as well as those groups where speaking and contributions were more evenly distributed. Finally, the studies found that collective intelligence positively correlated with the proportion of women in the group, most likely because women generally score higher in social perceptiveness tests.
Much additional research is needed, but there seems to be evidence that something like collective intelligence does indeed exist and can be measured. It’s primarily dependent on how well the individual members of the group work together and, to a lesser extent, on their individual abilities.
In another set of studies, Malone and collaborators looked at whether groups that included both humans and computers did better at making decisions than either the humans or computers by themselves. Their experiment, reported in this working paper, used the concept of prediction markets to predict what the next play would be in a football game. Some of the predictions were made by groups of humans; some by different computer-based statistical models; and some by combining the human and computer predictions.
They found that the computer-only predictions were better than those made by the human groups, but that the hybrid of humans and computers made the best overall predictions, being both more accurate and more robust to different kinds of errors. They attribute the results to the fact that people and computer models have different strengths and weaknesses. The computers use sophisticated statistical analysis and have no biases, but have trouble dealing with unstructured and common sense information that humans are very good at. On the other hand, humans are prone to biases and fatigue and are often not so good at evaluating probabilities. In addition, our judgement can be influenced by the dynamics of the group. “Therefore,” notes the paper, “combining human and machine predictions may help in overcoming the respective flaws of each.”
Malone’s talk, perhaps because it took place as the baseball playoffs were just getting underway, reminded me of the discussions on predicting the future performance of baseball players in Nate Silver’s The Signal and the Noise: Why Most Predictions Fail but Some Don’t. The book, published about a year ago, offers some of the best explanations I’ve read of the state-of-the-art of data-driven predictions in a variety of fields.
Silver gained national attention when he correctly predicted the results of the 2008 Democratic Party presidential primaries in his FiveThirtyEight website launched earlier that year. His final forecasts for the 2008 presidential elections predicted the winner in 49 of the 50 states, as well as the winner of every race in the Senate. His notoriety went even higher during the 2012 presidential elections, when he correctly predicted the winner in all 50 states, including all nine highly contested swing states, as well as in 31 of the 33 Senate races.
Silver learned his craft in the new field of sabermetrics, - the use of statistics in baseball to project a player’s performance and career. Sabermetrics was popularized by Michael Lewis in Moneyball, his bestseller book, - later turned into a film, - about Billy Beane, the Oakland Athletics general manager who used such statistical techniques to make his small-market team highly competitive against teams with much larger budgets. The Oakland A’s made the playoffs again this year, having won the American League West Division.
The Signal and the Noise devotes a whole chapter to sabermetrics. Sabermetrics is now widely used by every team in baseball. When Moneyball first came out in 2003, many viewed it as a story about the conflict between the traditional approach of the scouts, - the professional talent evaluators who learn about the players first-hand by meeting them in person and watching them play, - versus the new approaches being introduced by the statheads, - who mostly rely on sophisticated statistical analysis to predict future performance.
Years later, there was enough data to compare the performance of scouts versus more purely statistical approaches. The scouts’ predictions were about 15 percent better than those that relied on statistics alone. The good scouts, as it turns out, use a hybrid approach combining statistics with whatever else they learn about the players. Statistics alone cannot tell you everything you want to know about a player, and the additional personal evaluations of the scouts make a significant difference.
Billy Bean attributes the success of the Oakland A’s not just to their statistical aptitude but to their careful scouting of amateur players. Scouting
is particularly valuable when evaluating young amateur players while
they are still learning the game, because a good scout can spot certain
intangibles that may make a player stand out, such as their mental
makeup and overall attitude toward the game. He defines a good scout as someone who can find out information that other people can’t, such as getting to know the prospect and his family in person. In fact, their scouting budget is now much higher than it has ever been as a way of complementing their statistical analysis.
“In fact,” writes Silver, “the demand to know what the future holds for different types of baseball players, - whether couched in terms of scouting reports or statistical systems like PECOTA, - still greatly exceeds the supply. Millions of dollars, - and the outcome of future World Series - are put at stake each time a team decides which player to draft, whom to trade for, how much they should pay for a free agent.”
“Teams are increasingly using every tools at their disposal to make these decisions. The information revolution has lived up to its billing in baseball, even though it has been a letdown in so many other fields, because of the sports unique combination of rapidly developing technology, well-aligned incentives, tough competition, and rich data.”
The promise of cognitive systems and data science is that combining advanced technologies and lots of data with talented individuals and groups will help us better address the highly complex problems we face in business, government and society in general. The technologies will both help enhance the performance of individuals, as well as help groups work together more effectively. This combination of people and computers should hopefully allow us to take on some of our most difficult problems in ways that had never been possible before.
Malone concluded his talk with his personal observation of where this all might be leading us:
“As the world becomes more interconnected through the use of communications technology, it may become useful to view all the people and computers as part of a single global brain. It’s possible that the survival of our species will depend on combining human and machine intelligence to make choices that are not just smart but are also wise.”