A Second Opinion on Every AI Verdict: 17 Jev-Style Models Tested

Every scan that reaches our AI Security Verdict is read by a large language model. It weighs the evidence (forms, redirects, scripts, threat intelligence) and writes back a risk band, a 0–10 score, the evidence it relied on and a short explanation. A layer of deterministic safeguards then corrects the obvious mistakes: it lowers verdicts that over-call a legitimate site and raises ones that miss hard evidence.

The hottest thing in AI right now is Jev from TypeSafe AI, launched on 15 September. Jev is a typed-decision model. Instead of writing text, it answers a question in one of three shapes, yes/no, pick one of several options, or a score on a scale, with a calibrated probability, in a single fast pass. The response has been extraordinary. Within about a week, one community directory already listed 352 projects built on or around it, and more than 60 "Jev"-named models appeared on Hugging Face in just two days, many of them open attempts to reproduce the idea.

We wanted to know two things. Could one of them replace our language model? And if not, could one give every verdict a fast, cheap second opinion?

Why it cannot replace the language model

Half of what our AI verdict returns is written for people: the evidence list, the risk factors, the safety factors and the explanation you read on every scan page. Across three days of verdicts, 69–75% of those sentences were unique, because they cite the specific phone number, brand or hostname on the page in front of them. A model that only answers typed questions cannot write them.

So the question became where a typed model could check the language model rather than stand in for it.

The question we asked

We picked the decision where a mistake is most expensive. When our safeguards consider lowering an over-called verdict, they first ask: does this verdict assert a concrete malicious mechanism (credential harvesting, brand impersonation, a phishing kit, malware, data exfiltration, command-and-control), or does it deny one? If the verdict asserts one, the verdict must not be lowered.

Today a carefully tuned text matcher answers that question. The hard part is negation. Many high-risk verdicts say things like "no impersonation or malware was found, but the domain is three days old and the page asks for card details". A system that matches keywords sees "impersonation" and "malware". A system that reads sees that both are denied.

How we tested, and what we got wrong first

We built a test set from 30 days of real verdicts: cases our current matcher misses, cases it gets right because the text explicitly denies a mechanism, and random samples of both answers. Then we ran every candidate on it, on our own NVIDIA RTX 3080 (10 GB), with the same question worded the same way.

Our first benchmark was wrong in two ways. Both are worth sharing because they are easy to make.

1. We graded models on text the real check never sees. Some of our safeguards rewrite part of the verdict text after the evidence check has already run. We had built the test set from the stored, final text. So on 6.7% of the cases, including some of the hardest ones, models were reading sentences the production check never reads. We rebuilt the set from scratch, excluding every rewritten case.

2. Per-model thresholds faked the ranking. Our first scoring script let each model pick its own cut-off, so we were comparing eight different operating points. When every model had to catch the same known misses, the ranking turned upside down. Now we compare models at matched recall: set the threshold that catches every known miss, then count the false alarms.

The corrected set has 23 hard cases from 23 different sites, each reviewed three times independently, plus the known misses and 300 random verdicts. That is still small. It is enough to separate the models that work from the ones that don't, but not enough to rank the models in the middle against each other.

The models we evaluated

We tested 17 projects in 33 configurations: open, Jev-style replicas of the typed-decision idea, general language models used as judges, and plain natural-language-inference (NLI) models, which were built to decide whether one text supports, contradicts or ignores another.

Separation is how well a model ranks the real misses above the hard cases: 1.000 is perfect, 0.5 is a coin flip. False alarms is how many of the 23 hard cases it flags when set to catch every real miss.

Not in the table: minojev and Laya's multilingual version, tested on our earlier set, where both ranked random verdicts worse than a coin flip.

A fair reading of this table: it measures one narrow, unusual question on our data. Most of these projects are general-purpose and were days old when we tested them. Several were updated while we were testing them. None of these numbers says how good a model is in general.

What we learned

An off-the-shelf model won, with no training. The best result came from a general zero-shot NLI model asked one plain question. It ranked every real miss above every hard case, with a wide gap between the two groups, and answered in about 30 milliseconds on the RTX 3080. A trained decision model, decider-2b, matched it on this question, but by a much thinner margin.

Training on the wrong data is worse than no training. Four trained decision heads scored below chance or barely reacted at all. Each had been trained on other kinds of decisions (web navigation, general decision benchmarks, other languages) and did not carry over to reading security verdicts.

Negation in a question is not negation in a text. One model was trained specifically on negated questions. It ranked random verdicts best of all, yet flagged 22 of the 23 hard cases. Its training taught it to handle "is this NOT true?", but our problem is text that says "no malware was found".

One plain question beat several narrow ones. Splitting the question into six (one per mechanism) made results worse. Adding explicit true/false criteria hurt three of the four models that accept them.

The amount of text matters more than the choice of model. On the roughly 600 characters of a verdict's own explanation, the winning model also estimated the risk score closely (correlation 0.925 with our final score) and put 97.7% of verdicts within one band of the final one. On a page title of a few words, the same model mistook ordinary sites for security block pages 5.9% of the time. A typed decision works when it has a paragraph to read, not a phrase.

We did not test Jev itself. It runs only as a cloud service and is still in early access, and we don't send scanned page content to third parties for experiments. Everything above is open models running on our own hardware, which is also what lets us run them in production.

How we use it in production

The winning model and decider-2b now run side by side on a single NVIDIA RTX 3080 in a server we operate. Both fit in its 10 GB with room to spare, and answering all the questions for one verdict takes about 150 milliseconds. Since 23 September, for 10% of the verdicts our language model writes, both models read the model's own explanation, exactly as it was written and before our safeguards adjust it, and answer three questions:

• Does it assert a concrete malicious mechanism? (yes/no, both models)
• Which risk band does it describe? (one of five)
• What risk score does it describe? (0–10)

The answers are stored next to the verdict, and nothing acts on them yet. Three rules apply until the data says otherwise:

1. It never changes your verdict on its own. Your verdict is exactly the same with the second opinion on or off. Our test suite checks this in both modes.
2. Fail open. Each check has a two-second limit. A slow or unavailable model costs a data point, never a scan. Across three deployments on launch day, not one verdict was lost.
3. It can only add caution. If a check is ever allowed to act, it may only raise scrutiny. It may never lower a verdict or skip a scan. Scanned pages are written by attackers, so a second model gets more power to be careful, not more power to be lenient.

Before going live we replayed our full test sets through the production service. It reproduced the benchmark exactly, down to identical numbers on every case.

The first live results already show why a second opinion has to earn its say. One of the very first verdicts described a legitimate hotel brand's own website in careful, denial-heavy language: not impersonation, no malware, genuine brand material. The evidence question got it right; both models said no mechanism was asserted. The band question called it a confirmed scam. It had read the security vocabulary and missed the negation. Two of the first six live rows show the same pattern. That is too few to conclude anything yet, but it is exactly the kind of case a second opinion has to get right before it gets any say.