AI Empower: Democratizing AI – Empowering Individuals, Engaging Communities

System 2 Attention (is something you might need too)

Weston, J. and Sukhbaatar, S. (2023). System 2 Attention (is something you might need too). [online] Available at: [Accessed 9 Dec. 2023].

General Annotation #

System 2 Attention (S2A) addresses the challenge faced by Transformer-based Large Language Models (LLMs) where soft attention mechanisms may incorporate irrelevant information from the context into their latent representations, affecting next token generations. S2A is a technique that enables LLMs to first regenerate the input context to only include relevant portions before attending to this regenerated context to elicit a more accurate and relevant response. This process is designed to increase factuality, objectivity, and decrease sycophancy in model outputs.

Methodologies Used #

The implementation of S2A involves a two-step process where the model first regenerates a context (x') that only includes relevant parts of the original context (x). This is achieved by leveraging instruction-tuned LLMs to generate a prompt that guides the model to focus on relevant material. The regenerated context is then used for generating the final response. The study employs various implementations and variations of S2A, including experiments with and without context/question separation, the inclusion of original context, instructed prompting, and emphasizing relevance or irrelevance.

Key Contributions #

  • Introduction of S2A: A novel attention mechanism that improves LLMs’ ability to focus on relevant information and ignore spurious correlations and irrelevant context.
  • Experimental Validation: S2A outperforms standard attention mechanisms in LLMs across several tasks, including factual question answering, longform generation, and math word problems, demonstrating improvements in factuality, objectivity, and accuracy.
  • Ablation Studies: The paper presents various implementations and variations of S2A, providing insights into how different approaches affect the model’s performance.

Main Arguments #

The main arguments of the paper revolve around the inherent limitations of soft attention mechanisms in LLMs and the potential of S2A to mitigate these issues by leveraging the model’s natural language reasoning capabilities. The authors argue that S2A, by focusing on relevant context, can significantly enhance the model’s ability to generate accurate, factual, and unbiased responses.

Gaps #

While S2A shows promise, the technique has limitations, including the potential for the model to still be affected by spurious correlations. Additionally, the method requires more computation than standard LLM generation, and its effectiveness largely depends on the quality of the instruction-tuned LLMs and the prompts used.

Relevance to Prompt Engineering & Architecture #

S2A represents a significant advancement in prompt engineering and architecture, showcasing how tailored prompting mechanisms can direct LLMs to disregard irrelevant information and focus on the essence of the input. This technique underscores the importance of developing innovative prompting strategies to enhance the reasoning capabilities of LLMs and their application across various domains.

What are your feelings
Updated on March 31, 2024