LLMs Misdirect Users to Phishing Sites via Incorrect URLs
Research found that LLMs including GPT-4 frequently provide incorrect login URLs when queried, with over one-third of suggestions being wrong and many others pointing to unregistered or parked domains that attackers could exploit. In documented cases, the systems recommended phishing sites that closely mimic legitimate company branding, enabling credential theft at scale.
What happened
Research by Netcraft found that when queried for login URLs for 50 brands, an LLM returned 131 hostnames of which 34% were incorrect.
Of the suggested URLs that were correct, 29% led to unregistered, parked, or inactive domains that could be commandeered by attackers, and 5% pointed to unrelated legitimate businesses.
In one documented case, an AI-powered tool recommended a phishing website when a user asked for the Wells Fargo login page, suggesting a malicious Google Sites URL that closely mimicked Wells Fargo's branding.
Reported impact
Users could be misdirected to malicious websites when seeking login pages for legitimate companies, enabling phishing attacks and credential theft.
The misdirection exploits user trust in AI systems, which confidently provide incorrect answers, making attackers' ability to harvest credentials or distribute malware at scale more effective.
Index analysis
The inaccuracy stems from inherent limitations in LLM-generated content in identifying correct URLs, rather than from adversarial attacks or jailbreak attempts.
Attackers can exploit this vulnerability by registering domains and waiting for victims to land on them when recommended by AI systems, bypassing traditional security signals like verified domains or search rankings.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.