The Rise of AI-Assisted Spear Phishing: How Deceptive Language Models Automate Social Engineering
Computer programs created these fake messages using info anyone can find online, Newer research suggests these computer programs are learning to do more complicated tasks on their own.
When people talk about computer safety, they usually imagine dramatic movie scenes—like robots taking over or hackers breaking into top-secret servers. But there’s a simpler, more immediate problem: clever computer programs are getting really good at tricking people.
Jeremy Philip Galen and Richard Whaling explain that today’s software can look up information about you, figure out what you care about, pretend to be someone you trust, and hold a convincing conversation. This makes online scams much harder to spot than they used to be. Recent research shows that these new tools can make fake messages more realistic, more personal, and easier to send to many people at once [1].
Let’s talk about spear phishing. This is when scammers send fake messages to specific people, not just random strangers. Instead of guessing, they learn about you—your job, who you talk to at work, and even how you write emails.
There have been studies in which computer programs created these fake messages using information anyone can find online [2]. Imagine getting an urgent WhatsApp message from someone pretending to be your boss or your company’s security chief. If the message sounds real and believable, even smart, careful employees might click a bad link or share their passwords without thinking.
This is a big deal for businesses because most security disasters start with a person making a mistake or trusting the wrong message. Regular training on how to spot scams might not be enough anymore, since these computer-made messages are more convincing and keep getting better. Some studies show that these new scams can fool people just as easily as ones written by real humans—and they’re cheaper and faster to create [3].
Recent research suggests these computer programs are learning to perform more complex tasks on their own. There are even tests showing that scams could soon become fully automated, with computers selecting targets, gathering information, and sending out fake messages without any human involvement [4]. Experts think that by late 2026 or 2027, some of these attacks might run themselves from start to finish.
Some people think the current rules aren’t keeping up with these new threats. For example, California has a law that identifies scams as a major risk, but most places haven’t done much to stop large-scale fraud and trickery. Out of the big tech companies, only Meta (the company behind Facebook) treats huge scams as a top safety concern.
The authors suggest we need better tests and rules before these powerful programs are allowed out into the world, especially as they get smarter. This aligns with new efforts in cybersecurity, where experts are testing whether computers can write highly convincing fake messages [5].
The main warning here is simple: a huge disaster doesn’t always start with a hacker breaking into a computer. Sometimes all it takes is tricking someone into clicking a link or sharing their password. The authors say that regular people—not just big organizations—could become victims as these scams get more personal, harder to spot, and easier to believe.
References
[1] Oxford Internet Institute. “Digital Deception: Generative AI in Social Engineering and Phishing.” University of Oxford, 2024. https://ora.ox.ac.uk/objects/uuid:82397baa-d34e-46a1-9b90-e7f548c0a367
[2] Hazell, Julian. “Spear Phishing with Large Language Models.” arXiv, 2023. https://arxiv.org/abs/2305.06972
[3] Schneier, Bruce, et al. “Devising and Detecting Phishing Emails Using Large Language Models.” 2024. https://www.schneier.com/academic/archives/2024/03/devising-and-detecting-phishing-emails-using-large-language-models.html
[4] “Evaluating Large Language Models’ Capability to Launch Fully Automated Spear Phishing Campaigns.” arXiv/AlphaXiv, 2024. https://www.alphaxiv.org/abs/2412.00586
[5] Bhakta, Rajiv, et al. “CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models.” 2024. https://www.sciencestack.ai/paper/2408.01605
#ArtificialIntelligence #Cybersecurity #SpearPhishing #AISafety #SocialEngineering #OnlineScams