中文标题#
三元 ReLU 回归神经网络的线性区域数的下界
英文标题#
A Lower Bound for the Number of Linear Regions of Ternary ReLU Regression Neural Networks
中文摘要#
随着深度学习的发展，降低计算复杂度和内存消耗已成为一项关键挑战，限制参数为 {−1,0,+1} 的三元神经网络（NNs）作为一种有前景的方法引起了关注。虽然三元 NNs 在图像识别和自然语言处理等实际应用中表现出色，但其理论理解仍不够充分。在本文中，我们从线性区域数量的角度对三元 NNs 的表达能力进行了理论分析。具体而言，我们评估了使用修正线性单元（ReLU）作为激活函数的三元回归 NNs 的线性区域数量，并证明了线性区域的数量随网络宽度呈多项式增长，随深度呈指数增长，这与标准 NNs 相似。此外，我们表明，只需将三元 NNs 的宽度平方或深度加倍，就可以达到与一般 ReLU 回归 NNs 相当的最大线性区域数量的下界。这在某种程度上为三元 NNs 的实际成功提供了理论解释。
英文摘要#
With the advancement of deep learning, reducing computational complexity and memory consumption has become a critical challenge, and ternary neural networks (NNs) that restrict parameters to {−1,0,+1} have attracted attention as a promising approach. While ternary NNs demonstrate excellent performance in practical applications such as image recognition and natural language processing, their theoretical understanding remains insufficient. In this paper, we theoretically analyze the expressivity of ternary NNs from the perspective of the number of linear regions. Specifically, we evaluate the number of linear regions of ternary regression NNs with Rectified Linear Unit (ReLU) for activation functions and prove that the number of linear regions increases polynomially with respect to network width and exponentially with respect to depth, similar to standard NNs. Moreover, we show that it suffices to either square the width or double the depth of ternary NNs to achieve a lower bound on the maximum number of linear regions comparable to that of general ReLU regression NNs. This provides a theoretical explanation, in some sense, for the practical success of ternary NNs.
PDF 获取#
抖音扫码查看更多精彩内容