中文标题#
合成自適應引導嵌入（SAGE）：一種新型知識蒸餾方法
英文标题#
Synthetic Adaptive Guided Embeddings (SAGE): A Novel Knowledge Distillation Method
中文摘要#
模型蒸餾使知識從大規模模型轉移到緊湊的學生模型成為可能，有助於在資源受限環境中部署。 然而，傳統的蒸餾方法常常面臨計算開銷大和泛化能力有限的問題。 我們提出了一種新穎的自適應蒸餾框架，該框架在學生模型損失較高的區域動態增強訓練數據。 使用基於 UMAP 的降維和最近鄰採樣，我們的方法識別嵌入空間中表現不佳的區域，並生成有針對性的合成示例以指導學生學習。 為了進一步提高效率，我們引入了一個輕量級的教師 - 學生接口，繞過教師的輸入層，實現在向量化表示上的直接蒸餾。 在標準自然語言處理基準測試中的實驗表明，我們的 66M 參數學生模型始終與現有基線相當或超越，分別在 QNLI 上達到 91.2%，在 SST-2 上達到 92.3%，同時訓練的輪次更少。 這些結果突顯了基於損失的數據增強和向量化蒸餾在高效且有效的模型壓縮方面的潛力。
英文摘要#
Model distillation enables the transfer of knowledge from large-scale models to compact student models, facilitating deployment in resource-constrained environments. However, conventional distillation approaches often suffer from computational overhead and limited generalization. We propose a novel adaptive distillation framework that dynamically augments training data in regions of high student model loss. Using UMAP-based dimensionality reduction and nearest neighbor sampling, our method identifies underperforming regions in the embedding space and generates targeted synthetic examples to guide student learning. To further improve efficiency, we introduce a lightweight teacher-student interface that bypasses the teacher's input layer, enabling direct distillation on vectorized representations. Experiments across standard NLP benchmarks demonstrate that our 66M-parameter student model consistently matches or surpasses established baselines, achieving 91.2% on QNLI and 92.3% on SST-2, while training with fewer epochs. These results highlight the promise of loss-aware data augmentation and vectorized distillation for efficient and effective model compression.
文章页面#
PDF 獲取#
抖音掃碼查看更多精彩內容