When people hear "artificial intelligence," many envision "big data. " There's a reason for that: some of the most important AI breakthroughs in the past decade have relied on enormous data sets. But AI is not only about large data sets, and research in "small data" approaches has grown extensively over the past decade—with so-called transfer learning as an especially promising example. Also known as "fine-tuning," transfer learning is helpful in settings where you have little data on the task of interest but abundant data on a related problem. The way it works is that you first train a model using a big data set and then retrain slightly using a smaller data set related to your specific problem.
Research in transfer learning approaches has grown impressively over the past 10 years. In a new report for Georgetown University's Center for Security and Emerging Technology (CSET), we examined current and projected progress in scientific research across "small data" approaches. Our analysis found that transfer learning stands out as a category that has experienced the most consistent and highest research growth on average since 2010. This growth has even outpaced the larger and more established field of reinforcement learning, which in recent years has attracted widespread attention.
Small data approaches such as transfer learning offer numerous advantages over more data-intensive methods. By enabling the use of AI with less data, they can bolster progress in areas where little or no data exist, such as in forecasting natural disasters that occur relatively rarely or in predicting the risk of disease for a population set that does not have digital health records.
Another way of thinking about the value of transfer learning is in terms of generalization. A recurring challenge in the use of AI is that models need to "generalize" beyond their training data. Because transfer learning models work by transferring knowledge from one task to another, they are very helpful in improving generalization in the new task, even if only limited data were available.
Moreover, by using pretrained models, transfer learning can speed up training time and could also reduce the amount of computational resources needed to train algorithms (算法). This efficiency is significant, considering that the process of training one large neural (神经系统的) network requires considerable energy.
Despite the growth in research, transfer learning has received relatively little visibility. The existence of techniques such as transfer learning does not seem to have reached the awareness of the broader space of policy makers and business leaders in positions of making important decisions about AI funding and adoption. By acknowledging the success of small data techniques like transfer learning—and distributing resources to support their widespread use—we can help overcome some of the common misconceptions regarding the role of data in AI and facilitate innovation in new directions.
