Open Internet by MindsNet

Inter-layer parameter redundancy in Transformers

Existing Transformer models, including Dense and standard Mixture-of-Experts (MoE) models, suffer from inter-layer parameter redundancy. This redundancy leads to inefficient use of parameters and limited model capacity. The problem arises from the independent learning of parameters in each layer, resulting in duplicated efforts and wasted resources.

Computing & Technology, Computer Science, Artificial Intelligence