MIT researchers reveal LLMs learn incorrect syntactic patterns during training
MIT researchers found that large language models can learn incorrect grammatical patterns during training and rely on these patterns instead of reasoning, potentially producing unreliable answers in customer service, clinical documentation, and financial reporting tasks. The vulnerability could also be exploited by malicious actors to bypass model safeguards.
What happened
MIT researchers discovered that large language models can learn to mistakenly link certain sentence patterns with specific topics during training, then repeat these patterns instead of reasoning through problems.
An LLM given a question with the same grammatical structure as training examples but with nonsense words might provide a convincing answer by recognizing familiar sentence patterns instead of understanding the actual question.
The researchers found that even the most powerful LLMs can make this mistake, which they call reliance on incorrect 'syntactic templates' learned from parts-of-speech patterns in training data.
Reported impact
This shortcoming could reduce the reliability of LLMs performing tasks like handling customer inquiries, summarizing clinical notes, and generating financial reports.
A nefarious actor could exploit this vulnerability to trick LLMs into producing harmful content, even when the models have safeguards to prevent such responses.
Index analysis
The researchers developed a benchmarking procedure to evaluate a model's reliance on these incorrect correlations to help developers mitigate the problem before deploying LLMs.
Governance controls
The shortcoming is a byproduct of how models are currently trained, but the researchers note that models are now deployed in safety-critical domains far beyond the tasks that created these syntactic failure modes.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.