I often look to biology and evolution for inspiration when thinking about highly complex systems. So I was very intrigued when several weeks ago I learned about the work being done by Elemental Cognition, a company founded in 2015 by my former IBM colleague David Ferrucci. Ferrucci led the team that developed the Watson computer system, which in 2011 won the television quiz show Jeopardy! against two of the world’s top Jeopardy champions. Let me explain.
Large language models (LLMs), chatbots, and generative AI have introduced technologists like me to the fascinating world of human language and cognition. Last year, for example, I learned the difference between four important linguistic concepts in a 2020 research paper by linguistic professors Emily Bender and Alexander Koller.
- Form is any observable expression of language, whether written, spoken, or signed.
- Communicative intent is the purpose a speaker intends to achieve through language, such as to convey some information or just to socialize.
- Meaning is the relation between the form in which the language is expressed and the communicative intent it’s being used to evoke in the listener or reader.
- Understanding is the listener’s ability to capture the meaning that the speaker intends to convey.
The authors wrote that while the success of LLMs and chatbots on many natural language tasks is very exciting, “these successes sometimes lead to hype in which these models are being described as understanding language or capturing meaning.” However, “a system trained only on form has apriori no way to learn meaning.” Research on language acquisition has found that “human language learning is not only grounded in the physical world around us, but also in interaction with other people in that world. … Human children do not learn meaning from form alone and we should not expect machines to do so either.”
“Do We Need Language to Think?,” asked a recent NY Times article. “A group of neuroscientists argue that our words are primarily for communicating, not for reasoning,” the article adds, referencing a research paper published last month in Nature by cognitive neuroscientists Evelina Fedorenko, Steven Piantadosi, and Edward Gibson, “Language is primarily a tool for communication rather than thought,.”
Their paper argued that “in modern humans, language is a tool for communication, contrary to a prominent view that we use language for thinking.” Based on recent evidence from neuroscience and related disciplines, the authors concluded that “although the emergence of language has unquestionably transformed human culture, language does not appear to be a prerequisite for complex thought, including symbolic thought. Instead, language is a powerful tool for the transmission of cultural knowledge; it plausibly co-evolved with our thinking and reasoning capacities, and only reflects, rather than gives rise to, the signature sophistication of human cognition.”
Using brain scanning tools like functional MRI, cognitive neuroscientists have shown that human language processing, — including spoken, written or signed language, — is concentrated in specific brain areas in the frontal and temporal lobes, typically in the left hemisphere. The set of brain language areas is only used for language processing but are not engaged in non-language cognitive tasks like solving problems, doing arithmetic, listening to music, recognizing facial expressions, or thinking in general.
Are we expecting too much from LLMs, given that this is not how our human brains, which have evolved over tens of millions of years, actually work? We seem to be asking LLMs to handle both the formal rules, patterns, and vocabularies of language, as well as the reasoning and thinking needed to understand the meaning of what’s being said.
Elemental Cognition (EC) has developed what it calls the EC AI platform, whose architecture follows human biology by separating its natural language components from its reasoning, problem solving engine. The natural language components are used as the input/output interfaces for interacting with humans for knowledge acquisition and delivery. The reasoning engine combines multiple precise logical and mathematical methods to solve hard problems efficiently and transparently.
Ferrucci nicely explained the EC AI platform in a series of three blogs.
The Limitations of Large Language Models for Complex Reasoning. “From industry leaders to AI-curious individuals, many hold up Large Language Models (LLMs) as the answer to solving complex problems and making better decisions,” begins the first blog. “While they are powerful tools, it’s important to understand their limitations and use them effectively. Think of LLMs as skilled writers and storytellers, not as reasoning experts. They are amazing at understanding and generating human language, but they don’t necessarily understand the logic behind it.”
Ferrucci illustrated the limitations of LLMs by discussing the difference between natural and formal languages. Natural languages, — e.g.,English, Spanish, Chinese, — are ambiguous and imprecise. They evolved to enable humans to communicate with each other, but are insufficient for reliable logical reasoning and computations. Formal languages, like those used in math and programming, were created “to rigorously and explicitly apply clear, reliable rules of logical inference to draw conclusions and compute answers.”
“For complex reasoning problems where you cannot afford to be wrong, natural language is not the right medium. Without any underlying formalism, natural language’s ambiguity and subjectivity are great for casually navigating around into another human’s brain, but not the best for ensuring shared meaning and precise, reliable outcomes. It’s why we invented formal languages and reasoning systems, and those inventions enabled science, mathematics, and the technology revolution.”
Large Language Models Created Demand for AI Capable of Complex Reasoning They Can’t Deliver Alone. “How can we leverage the power of AI to solve complex problems while ensuring its insights are transparent, accountable, and accurate?,” asks Ferrucci in the second blog, “The answer lies in bridging the gap between natural language and formal reasoning.”
The blog explores in more detail why LLMs are incapable of delivering the reasoning capabilities that businesses expect AI to deliver. Relying on LLMs to solve hard business problems is dangerous, because complex reasoning often requires formal logic and mathematical algorithms.
“Humans effortlessly weave knowledge and meaning using natural language, relying on collective experiences and shared understandings. However, this inherent efficiency comes at a cost. Personal experiences, biases, and emotions infuse our words, blurring the edges of meaning and leading to divergent interpretations of the same text.”
While natural languages allow us to share information with remarkable efficiency, their ambiguity can lead to misunderstandings and hinder our ability to solve complex problems. “Reliable decision-making processes, like any formal reasoning, require precise semantics, or meaning. Systems like computer programming languages are formal languages. They use clear rules and have specific ways to get correct answers. There is a mathematically precise, objectively correct decision procedure to determine exactly how to interpret each expression, and how its meaning feeds into the meaning of the next expression.”
The LLM Sandwich: AI that Solves Complex Problems with Reliable Reasoning. “[H]ow can we build more holistic AI to help make reliable, accurate, and transparent decisions when the stakes are high?,” asked Ferrucci in the third and final blog in the series. “The answer lies in using LLMs to create fluent natural language interfaces to formal systems capable of complex reasoning.”
“Simple analogies are a great way to better understand technology, so that’s why we have coined this architecture the LLM Sandwich,” he added. LLMs are the bread and the reasoning engine is the filling. “LLMs play an essential role in this architecture. They unlock a huge opportunity to interact in natural language with a general purpose reasoning engine that is capable of reliable, interactive problem-solving using formal mathematical algorithms.”
“The crucial point is that the reasoning engine solves the problem, not the LLM. This is how we can satisfy the business need for AI to solve complex problems reliably, accurately, and transparently, while still making it easy to interact with AI.”
The EC AI platform offers four significant benefits:
User-friendly, reliable problem-solving. “Enabling people to interact directly with the formal reasoning engine in natural language means they can explore the trade-offs inherent in complex problems and make the optimal decision every time.”
Provably-correct answers. “EC AI’s reasoning engine uses mathematically precise, objectively correct decision procedures to determine exactly how to interpret each expression, and how the truth values of expressions affect one another.”
Total decision transparency. “This is not a black box generating answers using statistical analysis of word distributions. Results are predictable, repeatable, and there is a fully traceable decision logic you can see to justify every decision.”
Efficient run-time compute costs. “EC AI’s general purpose reasoning engine solves problems efficiently using rigorous reasoning that is not performed by LLMs. It can run on a phone with 8-24GB of RAM and scale easily on the cloud.”
The EC architecture is explained in detail in a comprehensive paper, “Beyond LLMs: Advancing the Landscape of Complex Reasoning.” The paper compared the effectiveness of the EC AI platforms to that of GPT-4 in three application domains: workforce, travel, and degree planning. “Our results show that across all three applications, the EC AI systems, with LLM-support for user interaction, were able to achieve 100% performance in all dimensions, significantly outperforming GPT-4, which was able to produce some valid solutions, the vast majority of which were suboptimal, and performed poorly in recognizing or rectifying invalid solutions.”
The performance advantage of the EC AI platform can be attributed to its distinctive approach of separating the mechanisms for user interaction and knowledge capture from the reasoning and problem solving mechanisms, said the paper in conclusion. “This clear distinction sets it apart as an AI platform, in contrast with LLMs that integrate them into an end-to-end framework.”
Comments