Merlin: 99.7% F1, but dependent on tool names

Merlin's near-perfect ASB score does not survive a change in tool names. We replaced names consistently while preserving the meaning of each invocation. Lowercasing leaves the unchanged model detecting only 22% of ASB attacks. Giving benign tools CamelCase names causes false-positive rates of 86% to 100%, depending on the test subset.

At a glance

• A rule that treats CamelCase as unsafe reaches 0.98 F1 on ASB without training.
• Merlin's original 0.997 F1 conceals substantial sensitivity to naming conventions.
• All measurements reported here come from our October 2026 evaluation. No attack payload was executed.

A tool name cannot stand in for a safety assessment

An assistant is asked to summarize an email. It proposes forwarding the message to an outside recipient instead. The relevant question is whether that action follows the user's request. Writing the tool name as send_email or SendEmail does not change the answer.

Tool-call classifiers are intended to assess a proposed invocation before execution. They need the request, preceding events, arguments and tool semantics. Names can supply useful hints. They cannot replace the relationship between the authorized task and the proposed action.

Merlin is a published local classifier with 70.8 million parameters and a 512-token input budget. Its model card reports 0.905 F1 across TS-Bench: 0.997 on ASB, 0.510 on AgentDojo and 0.694 on AgentHarm. It also acknowledges that ASB dominates the pooled result. Sources checked on October 15, 2026.

Our finding goes beyond the already disclosed weaker AgentDojo score: a superficial naming convention heavily influences the safety decision. The high original benchmark result does not adequately establish that the model understands whether an invocation is safe.

The dataset supplies a shortcut

ASB attack tools frequently have CamelCase names. Benign tools do not. Our evaluation found the following distribution:

The rule “CamelCase means unsafe” achieves 0.98 F1, 0.96 recall and 1.00 specificity on ASB. No transformer or training is needed.

A linear model using tool names alone reaches 0.97 AUROC on ASB. That demonstrates how strongly a superficial feature separates the labels in this dataset. Original AgentDojo and AgentHarm tool names contain no CamelCase.

Task overlap compounds the issue. Sixteen of ASB's 51 test tasks also occur in training, while ASB validation uses the same base tasks as the ASB training portion. A good validation result on that distribution does not reliably expose the shortcut.

Rename the tools and Merlin changes its verdict

We kept the published checkpoint unchanged and evaluated all 7,182 TS-Bench test rows at a fixed 0.5 threshold. Names were replaced consistently in history, actions and tool descriptions. Each invocation retained its meaning and correct label.

For ASB's 5,231 rows:

Some original values are rounded. Specificity is the fraction of benign calls correctly kept benign.

Lowercasing cuts recall to 22%. New CamelCase names produce the opposite failure: Merlin flags 86% of benign ASB calls. The same transformation yields 93% false positives on AgentDojo and 100% on AgentHarm.

On these modified inputs, Merlin fails in both directions. Naming style determines whether it misses attacks or blocks legitimate work. ASB balanced accuracy falls to between 0.57 and 0.69 across the variants. With CamelCase on AgentHarm it reaches 0.50 because every call is flagged.

Some ranking information survives in the scores. That does not resolve the operational failure: renaming shifts those scores enough that the published decision threshold no longer holds.

This is a substantial robustness defect in a safety classifier. A high original ASB result is an insufficient quality demonstration when a consistent renaming test exposes this behavior. Our criticism concerns the measured failure and its implications, not the authors' intentions.

Context only helps when it reaches the model

We also swapped the assessed tool's description with another description in the same row. None of Merlin's 5,231 ASB predictions changed. Roughly 5% changed on AgentDojo and 6% on AgentHarm.

This intervention changes input semantics, so the old label is not necessarily valid afterward. We therefore report prediction changes rather than accuracy. It establishes limited sensitivity to this particular swap, not that descriptions never influence Merlin.

There is also an input-budget problem. Merlin allocates 128 tokens to the schema. In our AgentDojo evaluation, the assessed tool's description remained visible in only 25% of unsafe rows and 56% of benign rows. Including a schema field does little if the relevant tool description gets truncated.

Our comparison models change the data and representation

An early model of our own trained on the same data learned the same shortcut: ASB F1 fell from 1.00 to 0.07 under snake_case renaming. The failure cannot be attributed solely to Merlin's architecture.

For subsequent internal experiments, we used 59,185 training rows, comprising 7,429 unsafe and 51,756 benign cases. Sources include TS-Bench training data, benign function-calling examples, additional attack data and synthetic workflows in ten languages. AgentDojo was not used for training, and test or validation text was not used as templates for synthetic examples.

Tool names vary across nine styles independently of the label. Other examples put the same action in authorized and unauthorized contexts. Additional language-model agents reviewed synthetic labels, and mislabeled cases were removed.

The revised representation keeps the assessed tool's description visible. Chain variants process calls and outputs separately in steps of up to 640 tokens and connect those steps through memory. We compare three mmBERT-small variants:

• A: the chain with tool outputs.
• B: the chain without tool outputs.
• C: a single invocation with its context.

Better on AgentDojo, worse on AgentHarm

F1 on the same original test subsets:

Pooling ASB and AgentDojo, C reaches 0.967 F1, 0.987 specificity and 0.951 recall. Merlin reaches 0.938, 0.950 and 0.937 respectively. ASB is much larger and strongly influences these aggregate results, which is why the breakdown matters.

AgentHarm also asks a different question. In the TS-Bench portion used here, labels depend heavily on whether the user's request is harmful. A request-only linear model cross-validated within AgentHarm reaches 0.98 F1. This is a different evaluation from the cross-source comparison with Merlin. It shows how much can be inferred from the request within that subset.

Our models target misuse of an invocation in context, and we did not optimize AgentHarm. Their lower scores belong in the results. They do not support a claim of overall superiority.

What this means for evaluation

A naming convention is not a dependable substitute for assessing tool-call safety. The renaming results expose how heavily Merlin's original performance relies on the dataset convention. Such interventions should be a basic requirement when evaluating a tool-call classifier.

Three checks are particularly useful: rename tools consistently, report every source separately and establish whether each label concerns the request or the proposed action. The latter two questions also apply to our comparison models.

Those experiments use one seed per variant. They do not establish a robust architectural ranking. We made training decisions, including removing certain request pairs after increased false positives, after inspecting TS-Bench results. The numbers describe an exploratory research process; a further untouched final test from real agent runs is missing. This qualification concerns the comparison models and does not change the Merlin renaming finding.

Wolf Defender keeps AgentDojo false alarms low

An independent study, dated October 2, 2026, compares Wolf Defender on benign tool outputs:

False-positive rates at the default 0.5 threshold. At a threshold set for 1% false positives, Wolf Defender detects 73.8% of AD-Matched and 83.9% of AD-Shift injections.

This study evaluates tool outputs and contains no direct comparison with Merlin.

Sources and evaluation scope

• Our measurements: saved October 2026 runs, prepared for this article on October 15. We used 7,182 TS-Bench test rows, Merlin revision 21b6f2cc, unchanged weights and a fixed 0.5 threshold. Renaming was consistent across input fields. Payloads remained inert text.
• Merlin: model card and reference implementation. Published original results and our modified-input results are identified separately.
• TS-Bench: ToolSafe repository and ToolSafe paper. Pinned data revisions are documented in Merlin's DATASETS.md.
• Internal comparison models: weights and training data are not released. Synthetic labels were reviewed by language-model agents rather than fully annotated by humans.

This article reports research, not general product performance. Please contact us with corrections to the measurements or their interpretation.