中文标题#
三元 ReLU 回归神经网络的线性区域数的下界
英文标题#
A Lower Bound for the Number of Linear Regions of Ternary ReLU Regression Neural Networks
中文摘要#
隨著深度學習的發展，降低計算複雜度和內存消耗已成為一項關鍵挑戰，限制參數為 {−1,0,+1} 的三元神經網絡（NNs）作為一種有前景的方法引起了關注。雖然三元 NNs 在圖像識別和自然語言處理等實際應用中表現出色，但其理論理解仍不夠充分。在本文中，我們從線性區域數量的角度對三元 NNs 的表達能力進行了理論分析。具體而言，我們評估了使用修正線性單元（ReLU）作為激活函數的三元回歸 NNs 的線性區域數量，並證明了線性區域的數量隨網絡寬度呈多項式增長，隨深度呈指數增長，這與標準 NNs 相似。此外，我們表明，只需將三元 NNs 的寬度平方或深度加倍，就可以達到與一般 ReLU 回歸 NNs 相當的最大線性區域數量的下界。這在某種程度上為三元 NNs 的實際成功提供了理論解釋。
英文摘要#
With the advancement of deep learning, reducing computational complexity and memory consumption has become a critical challenge, and ternary neural networks (NNs) that restrict parameters to {−1,0,+1} have attracted attention as a promising approach. While ternary NNs demonstrate excellent performance in practical applications such as image recognition and natural language processing, their theoretical understanding remains insufficient. In this paper, we theoretically analyze the expressivity of ternary NNs from the perspective of the number of linear regions. Specifically, we evaluate the number of linear regions of ternary regression NNs with Rectified Linear Unit (ReLU) for activation functions and prove that the number of linear regions increases polynomially with respect to network width and exponentially with respect to depth, similar to standard NNs. Moreover, we show that it suffices to either square the width or double the depth of ternary NNs to achieve a lower bound on the maximum number of linear regions comparable to that of general ReLU regression NNs. This provides a theoretical explanation, in some sense, for the practical success of ternary NNs.
PDF 获取#
抖音掃碼查看更多精彩內容