Social engineering in the age of generative AI: assess behaviour, not awareness

Social engineering is the manipulation of people into taking actions or disclosing information that compromises security. It does not exploit a bug in software; it exploits a feature of humans — we are helpful, we respect authority, we act under time pressure, and we follow processes even when the process leads somewhere unsafe. Year after year, breach reports find that the majority of incidents involve a human element. That number has not moved much in a decade, which tells you something about how most organisations have tried to address it.

This article was first published in early 2024 as an overview of tactics. It has been substantially reworked, because the ground shifted: generative AI changed both the attacker’s toolkit and the defender’s assumptions.

The tactics, briefly

The vocabulary is still useful, because the psychology underneath has not changed.

Phishing — fraudulent messages, usually email, designed to harvest credentials or deliver malware. Spear phishing targets a specific person with personalised context. Whaling targets executives.

Vishing and smishing — the same by phone and text message. Phone attacks are particularly effective because a voice creates urgency and rapport that text cannot.

Pretexting — inventing a scenario (“I am from the auditor, we need the export by noon”) that makes the request seem legitimate. The pretext often includes accurate details harvested from public sources.

Baiting and quid pro quo — offering something (a file, a favour, a service) in exchange for the target’s action.

Tailgating — physically following an authorised person through a controlled door. Still remarkably effective in organisations that consider themselves digital-first.

Underneath all of them are a handful of persuasion levers that Cialdini described decades ago: authority, urgency and scarcity, reciprocity, social proof, liking, and commitment. An effective attack stacks several of them.

What generative AI changed

For a decade, awareness training taught people to look for tells: awkward phrasing, wrong logos, generic greetings, odd sender addresses. Those tells were an artefact of attackers working at scale in a language they did not master. Generative AI removed them.

• Language is flawless and localised. A spear-phishing email in idiomatic German, matching the tone of the target’s own company, costs nothing to produce.
• Voices are clonable. A few seconds of audio from a conference talk is enough to place a call that sounds like the CFO. A widely reported case in 2024 involved an employee transferring a very large sum after a video call in which every other participant was a deepfake.
• Pretexts are researched automatically. Public information — org charts, LinkedIn activity, press releases, procurement notices — is compiled into a convincing scenario in minutes.
• Attack volume is unconstrained. Personalised attacks used to be reserved for high-value targets. Now everyone is a high-value target, because personalisation is free.

There is a second, quieter change on the defender’s side. Organisations have put AI assistants inside their workflows: coding agents with repository access, support bots with customer data, document assistants over internal knowledge. These are new social-engineering surfaces. An engineer pastes credentials into an assistant to “fix a config”. An agent follows an instruction embedded in a manipulated document. A support bot is talked into revealing account details through a plausible story. The target of manipulation is no longer only the human.

Why awareness training does not move the numbers

Most organisations respond with more training: an annual e-learning, a phishing simulation with a click rate, a poster. The problem is that awareness is not behaviour. In behaviour assessments I regularly meet people who can recite every rule and who still clicked — because:

• The message arrived at the busiest moment of the week and the request was consistent with their role.
• Responsibility was ambiguous: nobody was clearly accountable for verifying the request, so everyone assumed someone else had.
• Verifying was expensive: calling back meant finding a number, waiting, explaining, being seen as difficult.
• The tooling punished caution: reporting a suspicious message took six clicks and produced no feedback.

None of these are knowledge problems. They are design problems and culture problems, and they are specific to each organisation.

Assess before you train

My approach borrows from human-factors research on security-critical systems. Before designing any intervention, understand the actual behaviour:

1. Observe. Contextual observation of how people handle requests, approvals and exceptions — at their desks, under real load.
2. Interview. Structured conversations about the last time someone bypassed a control and why. People are remarkably candid when the aim is understanding, not blame.
3. Simulate. Controlled scenarios — email, phone, and increasingly AI-generated voice and video — designed around the organisation’s real processes, measured not only by click rate but by what happened next: did anyone verify, report, escalate?
4. Analyse the workflow. Where does the process itself make the unsafe action the easy one? Which approvals have no verification step? Which tools make reporting slow?

The result is a map of risk patterns with root causes, and it is almost never “people need more awareness”.

Design the workflow, then train for it

Interventions that move the numbers tend to look like this:

• Verification built into the process. Payment changes, credential resets and data exports require an out-of-band confirmation through a known channel — and the process makes that confirmation one step, not six.
• Explicit ownership. For each high-risk request type, one role is accountable for verification. Ambiguity is the attacker’s friend.
• Reporting that is instant and rewarded. One click, immediate acknowledgement, visible follow-up. People report when reporting is cheap and appreciated.
• AI assistants with designed limits. Least-privilege tool access, human confirmation for consequential actions, and input handling that treats retrieved documents as untrusted.

Training then teaches the designed workflow — to the actual teams, in their language, with scenarios from their work — and is measured with follow-up simulations. Interdisciplinary and intercultural teams need particular care: authority and directness are read differently across cultures, and a scenario that is obviously suspicious to one group is entirely plausible to another.

The number that should change

If your security programme reports a phishing click rate, add two metrics: the verification rate (how many risky requests were checked through a known channel) and the reporting time (how quickly a suspicious contact reached someone who could act). Those are behaviours. When they improve, the organisation is more resilient — regardless of how good the attacker’s language model is.