Open Internet by MindsNet

Preventing Attention Sinks without Positional Encoding

Removing positional encoding from self or cross-attention in transformer models leads to attention sinks, where query vectors attend to the same key tokens. This issue hinders dynamic attention based on query tokens. The challenge is to find a solution that enables query-conditioned attention without positional encoding.

Computing & Technology, Computer Science, Machine Learning