Introduction:

In the rapidly evolving field of natural language processing, large language models (LLMs) have demonstrated exceptional performance on various tasks. However, the computational demands of these LLMs can be prohibitive, hindering their accessibility and practical use. Knowledge Distillation (KD) has emerged as a promising technique to address this challenge by distilling knowledge from LLMs into smaller, more efficient models. While previous KD methods have been effective for classification models or imitating black-box model APIs like ChatGPT, the process of effectively distilling knowledge from white-box generative LLMs remains underexplored. In this article, we present MiniLLM, a novel approach to distilling smaller language models from generative larger language models, with improved precision, quality, and scalability.

The Need for Knowledge Distillation in Large Language Models:

Large language models have demonstrated remarkable language generation capabilities across a wide range of tasks. However, these models often require significant computational resources, making them less accessible to many researchers and developers. Knowledge Distillation offers a solution by compressing the knowledge from these large models into smaller, more efficient ones. This process not only reduces the computational demand but also allows the distilled models to achieve competitive performance with the larger counterparts.

Introducing MiniLLM: Distilling Smaller Models from Generative LLMs:

In our proposed approach, MiniLLM, we aim to effectively distill knowledge from white-box generative LLMs into smaller models. The key challenge lies in replacing the forward Kullback-Leibler divergence (KLD) objective, commonly used in standard KD approaches, with reverse KLD. The reverse KLD is better suited for KD on generative language models as it prevents the student model from overestimating the low-probability regions of the teacher distribution. This adjustment leads to more precise responses and higher overall quality in the distilled models.

Optimizing the Reverse Kullback-Leibler Divergence Objective:

Deriving an effective optimization approach is crucial for successful knowledge distillation. In MiniLLM, we fine-tune the reverse KLD objective to ensure efficient learning. This optimization process enhances the performance of the smaller language models, making them capable of generating high-quality responses and reducing exposure bias.

Experimental Results and Benefits of MiniLLM:

We conducted extensive experiments in an instruction-following setting to evaluate the performance of MiniLLM models. The results were highly promising, showcasing the superiority of MiniLLM in various aspects. The distilled models demonstrated more precise responses, lower exposure bias, improved calibration, and enhanced long-text generation performance. These findings highlight the effectiveness of our approach in achieving high-quality, efficient language models.

Scalability and Model Families:

MiniLLM is designed to be scalable and adaptable to different model families. With parameters ranging from 120M to 13B, our approach can distill smaller models effectively across a wide spectrum of language models. This scalability further enhances the practicality and versatility of MiniLLM in real-world applications.

Conclusion and Future Releases:

In conclusion, MiniLLM presents a groundbreaking approach to knowledge distillation of large language models. By leveraging the reverse Kullback-Leibler divergence objective, our method effectively distills knowledge from generative LLMs, producing smaller models with improved precision and quality. The promising experimental results demonstrate the potential of MiniLLM in transforming language processing for various applications. We are committed to further research and development in this domain, and we will release our code and model checkpoints to facilitate its adoption and integration into the NLP community. With MiniLLM, the future of efficient and high-performance language models is within reach.