Phishing Simulation Programs: What They Actually Measure

A phishing simulation program that shows a steadily declining click rate over successive campaigns looks, on the surface, like clear evidence of genuine security improvement, and it gets reported that way in plenty of security update meetings. What a falling click rate actually measures is considerably narrower than genuine phishing readiness, and businesses that treat it as the full story tend to develop a false sense of confidence about how their employees would actually respond to a real, sophisticated attack, as opposed to a simulated one employees have learned to recognize through repeated exposure to the same testing program.

Employees Learn the Simulation, Not Necessarily the Underlying Skill

A meaningful share of the improvement in click rates over time reflects employees learning to recognize the specific patterns a particular simulation vendor or internal program tends to use — a certain sender format, a certain subject line style, a certain visual tell — rather than genuinely improving their broader ability to recognize phishing attempts they haven’t specifically seen a version of before. This distinction matters enormously, because a real attacker isn’t constrained to the same patterns a simulation program reuses, and an employee who’s become excellent at spotting the simulation’s specific tells can still be genuinely vulnerable to a real attack that doesn’t happen to share those same recognizable characteristics.

The Gap Between Recognizing an Email and Actually Reporting It

Click rate measures one specific failure mode — did someone click a malicious link — but says nothing about a genuinely important complementary behavior: did the employee who correctly avoided clicking actually report the suspicious email to security, giving the organization a chance to warn others and investigate further. An employee who quietly deletes a suspicious email without reporting it scores perfectly on click rate while providing zero additional organizational value, and a program that only measures clicks has no visibility into this considerably more useful reporting behavior at all.

Repetition Effects That Inflate Apparent Improvement

Running simulations frequently against the same population inevitably produces some genuine improvement from repetition alone, simply because people become more generally vigilant when they know testing is an ongoing, recurring possibility, a phenomenon that has more to do with sustained alertness during a known testing period than with a durable, transferable improvement in actual phishing detection skill. This repetition effect can inflate the apparent success of a program in ways that don’t necessarily hold up during a genuine, unexpected attack that arrives without the heightened vigilance a known testing cadence tends to produce.

Difficulty Calibration That Quietly Keeps Success Rates Flattering

Simulation programs that want to show consistent improvement over time have a subtle incentive to keep difficulty roughly calibrated to what the current employee population can already handle, rather than genuinely escalating toward the sophistication of real-world attacks, which keeps click rates trending in a flattering direction without genuinely testing whether employees could handle a meaningfully more sophisticated, realistic attempt. A program that never meaningfully increases difficulty over time is measuring something closer to consistency against a fixed bar than genuine readiness against an evolving real-world threat landscape.

The Employees Who Never Get Caught by Any Simulation

Click rate is typically reported as an aggregate percentage across the whole organization, which can mask a genuinely important detail: a small subset of employees who fail every single simulation repeatedly, regardless of difficulty or format, representing a concentrated and disproportionate risk that an aggregate improving trend line can easily obscure. Identifying and providing targeted additional support to this specific group matters considerably more for actual organizational risk reduction than the overall aggregate trend, since a real attacker only needs one successful click, not an average success rate across the whole employee population.

Punitive Responses to Failed Simulations Can Backfire Quietly

Some organizations respond to a failed simulation with visible consequences — mandatory additional training, a note in a performance file — and while accountability has a genuine place, an overly punitive response can create a quiet, unintended incentive for employees to avoid reporting genuine suspicious emails they’re unsure about, out of fear that reporting something that turns out to be an internal test will itself trigger scrutiny. A security culture that feels punitive tends to suppress the very reporting behavior a healthy phishing defense program most needs to encourage, even while it succeeds in reducing the narrowly measured click rate.

Measuring Time-to-Report as a More Meaningful Indicator

A genuinely more useful metric than click rate alone is time-to-report — how quickly, on average, employees who spot something suspicious actually flag it to security — because this metric captures both genuine detection skill and the reporting behavior that gives an organization real, actionable early warning during an actual attack. A program that improves time-to-report meaningfully is building something closer to real organizational resilience than one that only improves click rate, since a fast, well-reported near miss protects the whole organization in a way that a merely avoided click, quietly unreported, never does.

Combining Simulation Data With Genuine Incident Response Testing

Phishing simulations measure a narrow slice of the overall security picture — whether an individual employee recognizes and avoids a specific malicious email — and pairing that data with broader incident response testing, including how quickly the security team itself detects and contains a genuine compromise once one has occurred, gives a considerably more complete and honest picture of organizational readiness than click rate data considered entirely in isolation. An organization can have an impressively low simulated click rate and still have genuinely weak incident response capability if a real compromise does eventually occur despite the strong simulation performance.

Varying Simulation Formats Beyond Just Email

Real phishing attempts increasingly arrive through channels beyond email — a text message impersonating a delivery service, a phone call impersonating internal IT, a message through a collaboration platform employees use daily — and a simulation program that only ever tests email leaves an organization with genuinely no data about how employees would respond to these other, increasingly common attack channels. Expanding simulation formats to reflect this broader real-world threat landscape takes more coordination to build and run than a single-channel program, but it closes a meaningful blind spot that an email-only click rate, however carefully tracked, simply cannot see into on its own.

Sharing Results Transparently Instead of Treating Them as Confidential

Some organizations keep simulation results tightly confidential, visible only to security leadership, which limits the program’s value primarily to a compliance metric rather than a genuine organizational learning tool. Sharing aggregate results transparently with the broader employee population, including honest examples of what made a particular simulation convincing and what specific details should have raised suspicion, turns each simulation into a genuine shared learning moment rather than a private test whose lessons stay locked away with the security team that ran it.

A Declining Click Rate Is a Data Point, Not a Verdict

Phishing simulation programs are genuinely valuable tools, but the specific metric most commonly reported from them — the click rate — measures something narrower and more susceptible to gaming and repetition effects than the phrase “phishing readiness” tends to suggest in a summary slide. Businesses that pair click rate with reporting behavior, time-to-report, identification of consistently vulnerable individuals, and periodic genuine escalation in simulation difficulty get a meaningfully more honest picture of their real exposure. Businesses that track click rate alone risk mistaking a population that’s learned to recognize a specific recurring test for a population that’s genuinely prepared to recognize whatever a real, determined attacker eventually sends instead.