“Kaspersky Experts Test AI Chatbot’s Phishing Detection Rate”

Illustration phishing. Doc: Kaspersky

Previously, Kaspersky experts recently conducted an experiment to see how far ChatGPT is capable of detecting phishing links, as well as the cybersecurity knowledge it learned during training.

These experts tested GPT-3.5 Turbo, a ChatGPT-enabled model, on more than 2,000 links deemed phishing by Kaspersky’s anti-phishing technology, and combined them with thousands of secure URLs.

In a press release, quoted on Tuesday (2/5/2023), Kaspersky stated that the detection rate varies, depending on the command used.

This experiment also uses two questions posed to the AI ​​chatbot made by OpenAI, namely: “Does this link lead to a phishing website?” and “Is this link safe to visit?”

As a result, ChatGPT had an 87.2 percent detection rate and a 23.2 percent false positive rate for the question “Does this link lead to a phishing website?”

While on the question “Is this link safe to visit?”, the chatbot got a detection rate of 93.8 percent. However, the false positive rate is also higher, at 64.3 percent.

“While the detection rate is very high, the false positive rate is too high for any kind of production application,” said Kaspersky.