Artificial intelligence first came to light in the mid-1950s as a promising new academic discipline. AI became one of the most exciting areas in computer sciences over the next two decades. But, after years of unfulfilled promises and hype, a couple of so called AI winters of reduced interest and funding set in that nearly killed the field. AI was successfully reborn in the 1990s with a new statistical paradigm based on analyzing large amounts of data with powerful computers and sophisticated algorithms. Now, six decades after the field was founded, AI seems to be finally coming of age.
“2021 saw the globalization and industrialization of AI intensify, while the ethical and regulatory issues of these technologies multiplied,” said the 2022 AI Index report on the progress of AI, which was released in March of 2022 by Stanford’s Institute for Human-Centered Artificial Intelligence (HAI). “2021 was the year that AI went from an emerging technology to a mature technology - we’re no longer dealing with a speculative part of scientific research, but instead something that has real-world impact, both positive and negative,” wrote Jack Clark, co-chair of the AI Index. Multiple factors led to his conclusion, in particular the advent of foundation models like OpenAI’s GPT-3 and Google’s BERT.
In the past decade, increasingly powerful AI systems have matched or surpassed human levels of performance in a number of specific tasks like image and speech recognition. These task-specific deep learning (DL) systems have generally relied on supervised learning, a training method where the data must be carefully labelled, - e.g., cat, not-cat, - thus requiring a big investment in time and money to produce a model that’s narrowly focused on one task and can’t be easily repurposed.
Foundation models have gotten around these DL limitations based on two recent advances, transfer learning and huge scale. Transfer learning takes the knowledge learned from training a specific task and applies it to different but related tasks, - such as using the training in recognizing cars in images and applying it to recognizing trucks and buses. As a result, foundation models can be adapted to many different tasks with relatively small amounts of fine-tuning.