How We Run Real-Time AI Security Inference On-Device on a CPU

Part I of our series on building Patronus Ark for CPU-based AI security inference.

Most AI security inference is designed as if a cloud service or a free GPU were always available. We wanted to test a different assumption: a complete security stack should be able to run continuously where AI interactions actually happen, on the endpoint, on an ordinary CPU.

Patronus Ark now runs inside Patronus Protect Desktop on a 2021 M1 MacBook Air with 16 GB of RAM. Without GPU optimization, the whole Desktop application peaks at roughly 750 MB RSS and performs its event-dependent security analysis at around 500 milliseconds median latency.

This is not a benchmark of one small classifier against one short sentence. Inputs range from a single 256-token chunk to roughly 30 to 40k tokens. Depending on the event and what earlier layers detect, Ark executes different categories and different depths of analysis. The current stack can detect prompt injections and threats, classify tools and sensitive documents, and extract dynamic PII.

The important result is not one isolated latency number. It is that several AI security tasks can run together, continuously, on a laptop CPU while coding assistants are actively working.

Why should AI security run on a CPU?

LLMs already consume the expensive hardware.

If every security decision requires another GPU, or a round trip to a large cloud model, the protection layer starts competing with the system it is supposed to protect. That is especially unattractive on endpoints, private infrastructure, and high-volume scanning APIs.

The privacy argument is the obvious one. Prompts, files, retrieved web content, tool calls, and model responses can contain source code, credentials, personal data, or internal documents. If the decision happens locally, none of that has to be forwarded to another inference service just to decide whether it is safe.

But privacy was not the only reason. Enforcement already happens on the endpoint. If an agent is about to call a tool, upload a document, or follow an instruction hidden inside retrieved content, the security result is needed before the action continues. Sending the complete interaction somewhere else first adds another dependency to the exact point where the system is supposed to intervene.

Local inference also keeps that decision available without a network connection. And once the same engine runs efficiently on a CPU, it can move between laptops, conventional servers, and private infrastructure without changing the basic architecture.

That last point changes the economics as well. We are testing whether the same work can make high-volume prompt-injection and content scanning APIs substantially cheaper. Long inputs, especially around 50k tokens, still need more throughput work, so this is not a result we want to claim yet. But the remaining problem is CPU scaling, not access to more GPUs.

The constraints became the architecture

We started with a fairly unforgiving product requirement: Patronus had to bring AI security onto the endpoint. That meant more than making inference work on our development machines. The stack had to remain usable on older business laptops with 8 GB of RAM, without a GPU, and it had to work well for German organizations.

Most competing systems did not have to accept those constraints. Cloud inference can add another model, another GPU worker, or more memory. On an endpoint, every additional model competes with the user's browser, IDE, office applications, local agents, and sometimes the LLM itself.

Our goal was therefore never to shrink one prompt-injection classifier and call it an endpoint security stack.

Ark has to make several different decisions. It needs to recognize injections and security threats, understand what kind of tool an agent wants to use, identify sensitive document classes, and find dynamic entities such as PII. It must handle short prompts as well as long files and traces. It must also stay responsive while other applications are already using the CPU.

It gave us a fairly strict set of constraints:

• CPU-only execution;
• less than roughly 1 GB of peak memory for the complete runtime;
• bounded processing of long and highly variable inputs;
• several security tasks without loading a separate encoder for each one;
• no repeated tokenization and embedding between the cheap and expensive decision paths;
• and deeper inference only where it actually changes the result.

Running every model for every chunk was not an option. Neither was loading a different transformer for every security category.

The first versions made that very obvious. A stack with separate models was moving toward roughly 3 GB of memory, already too much for the machines we actually wanted to protect. The current Desktop runtime peaks at roughly 750 MB. Reaching that point required us to solve several different problems rather than finding one lucky quantization setting.

What is Patronus Ark?

Patronus Ark is our layered AI security inference engine. It processes text, files, AI interactions, and agent-related content, and turns them into security evidence that an application can use for policy and enforcement.

The current classification pipeline has three layers.

Layer 1 currently contains native regular-expression heuristics. We are extending this into more structured recognizers built around anchors, candidate values, context, and validation. This is useful for signals that have a known structure: credentials, identifiers, explicit secrets, and other cases where a deterministic check is more appropriate than a semantic model.

Layer 2 consists of our Non-Transformer Decision Blocks, or NTDBs. An NTDB combines several inexpensive learned classifiers, including small neural heads, LightGBM, and logistic regression, with attention over one or more chunks. It produces the initial learned decision and the information needed by a Promote Gate.

Layer 3 is Lion Warden, our unified mmBERT security classifier. It provides the deeper contextual analysis, but it does not run automatically for every chunk.

Dynamic PII currently remains a separate path. We use a quantized GLiNER model to identify context-dependent entities and return their exact spans. This component accounts for a relevant part of the current memory footprint and is one of the next areas we want to integrate more tightly with the shared architecture.

One representation, two decision depths

This is the part that makes the current system substantially different from placing a small classifier in front of a completely separate transformer.

NTDB and Lion Warden share the same mmBERT tokenizer, embedder, and latent space.

For each input, Ark splits the text into chunks of up to 256 tokens. Tokenization and embedding happen once. The NTDB classifiers then operate on this shared representation and make an inexpensive Layer-2 decision.

A learned Promote Gate evaluates whether that decision is sufficient. If it is, the chunk stops at Layer 2. If deeper contextual processing is expected to help, the same chunk continues through Lion Warden's mmBERT layers.

In simplified form:

text
  → tokenize once
  → embed once
  → shared latent representation
      ├─ NTDB decision → stop at L2
      └─ Promote Gate → continue through mmBERT

This saves more than one model call.

We do not need a second tokenizer and a second embedding table for the deeper path. We do not calculate the initial representation twice. Layer 2 and Layer 3 also make their decisions in the same latent space, which gives the Promote Gate a more useful basis for deciding whether additional mmBERT layers are likely to improve the result.

Lion Warden itself is unified across several tasks. Instead of loading a separate transformer for injection detection, sensitive documents, tool classification, and threat detection, one shared mmBERT stack exposes the relevant security heads.

The dominant representation cost is therefore shared twice: across decision depths and across security tasks.

Selective depth already removes most Full-L3 executions

The Promote Gate is not a fixed confidence threshold. Layer 3 is not always right just because it is larger, and Layer 2 is not always wrong when it is uncertain. The gate learns when a Full-L3 execution is likely to rescue a Layer-2 mistake, and when it would merely repeat or even overwrite a correct decision.

Across the eight evaluation pipelines we currently consider reliable enough to report, roughly 16 to 24% are promoted at the evaluated operating points. In other words, Ark avoids approximately 76 to 84% of Full-L3 executions.

Five of those eight pipelines remain within one F1 percentage point of always running Full L3: Injection, Threat, Source Sensitive, Source Untrusted, and Sink External.

Sensitive Documents, Tool Class, and Tool Action are not yet within that margin. We are not hiding that difference: selective inference creates a task-dependent quality to compute curve, not automatic parity at every promotion rate.

The full per-pipeline evaluation and the idea of overlapping Layer-2 and Layer-3 decision spaces belong in Part III of this series. For now, the important point is that the system can remove most contextual transformer executions without treating the Layer-2 model as a disposable draft.

What the 500 ms median represents

It would be easy to put the fastest per-chunk number in the headline. It would also be misleading.

Layer 2 can make individual decisions much faster than the complete Desktop observation, and a Full-L3 execution is more expensive. But Patronus Protect is not processing one fixed benchmark string in isolation. It observes real AI-agent activity, selects the relevant security categories, processes inputs ranging from one chunk to tens of thousands of tokens, promotes selected chunks, runs dynamic PII where required, and aggregates the results.

The approximately 500 ms figure is the median for this event-dependent analysis in the live Desktop runtime. The approximately 750 MB figure is the peak RSS of the whole application across runtime. They are system-level measurements, not serialized model sizes or idealized single-head inference times.

That distinction matters because on-device inference succeeds or fails as a complete runtime. A 100 MB ONNX file is not useful if the loaded session, tokenizer, activation buffers, entity model, and host application consume several gigabytes. Likewise, a fast short-text benchmark says little about a security layer that must scan a long retrieved page or local file.

How we got from roughly 3 GB to 750 MB

The architecture above was not the starting point. Each part exists because an earlier version hit a concrete endpoint limit.

German security models and German security data

The German market could not be an afterthought. We needed models that understood German attacks, normal German business language, mixed German-English prompts, documents, and agent interactions.

Open-weight models that support German do exist. Meta's Prompt Guard 2, for example, reports German evaluation alongside several other languages. What we could not find was an open-weight, German-prioritized stack covering the broader endpoint problem we had: injections, threats, tool behavior, sensitive documents, and data exposure in one local architecture.

So we built the models and much of the data ourselves. That meant collecting public datasets, removing duplicates and leakage, correcting labels, generating missing German cases, translating and augmenting attacks without turning the dataset into translation noise, and adding hard benign examples that look security-related but are not attacks. Our open Wolf Defender model is one visible result of that work, but the same problem exists across the other security heads as well.

Fitting the stack into one gigabyte

An 8 GB endpoint does not have three spare gigabytes for a background security service. We set an upper budget of roughly 1 GB for the complete runtime.

Serialized model size alone does not solve that. We had to quantize the models, embeddings, and ONNX graphs, then measure what actually remained resident after tokenizers, inference sessions, activation buffers, the entity model, and the Desktop runtime were loaded.

Running many classifications without many encoders

Prompt injection was only one decision. Loading another transformer for threats, tools, routing properties, and sensitive documents would immediately destroy the memory budget.

Lion Warden therefore became a unified quantized classifier: one mmBERT representation with several task-specific security heads. Adding another classification task no longer means adding another full encoder.

Meeting an interactive latency budget

Even one shared transformer is too expensive if it runs over every chunk of every event. This led to the NTDB architecture and the learned Promote Gate.

NTDB makes the cheap decision first. The gate decides whether continuing through the contextual mmBERT layers is likely to improve it. We are not merely executing a smaller model, we are learning how much of the model a particular chunk needs.

Making room for dynamic PII

GLiNER gives us something a normal classifier cannot: dynamic entity types and exact spans. It also adds another substantial model to the process.

Sharing the mmBERT tokenizer, embedder, and latent space between Layer 2 and Layer 3 removed duplicated representation costs and brought the full Desktop stack down from roughly 3 GB to around 750 MB peak RSS, including the current separate quantized GLiNER path.

The result is the layered system we use today. For the first time, we have a build that does more than satisfy the original endpoint constraint. It suggests that the architecture forced by those constraints can become competitive with, and in some deployment settings better than, an always-on GPU inference design.

The open problems

The current system is practical, but three large pieces of work remain.

Bring GLiNER into the shared mmBERT architecture

Our dynamic PII model is currently based on DeBERTa. Training the GLiNER architecture on mmBERT would keep the latent space consistent across the complete stack and remove another separately loaded encoder. The goal is not a short-lived low-memory benchmark, but a stable peak RSS around 500 MB under real use.

Early internal ablations also indicate that NTDBs can pre-decide which chunks need full GLiNER entity extraction. Dynamic PII would then follow the same rule as the rest of Ark: execute the expensive contextual path where it adds information, not over every chunk.

Cascade the Layer-3 execution itself

Today, the main Promote Gate decides whether a chunk enters Lion Warden. We are now adding NTDB-style decisions inside Lion Warden's 22 mmBERT layers.

In the first experiments, around 90% of cases can be decided before layer 12, while fewer than 1% need the complete 22-layer execution. The early latency results suggest that this could at least halve p99 latency and reduce p50 by roughly one quarter. These numbers still need full ablation.

If they hold, Ark becomes much more interesting as a CPU-based API engine as well. Several CPU containers could process the same security workload at a fraction of current GPU inference costs while approaching similar execution times. That is the direction; the long-input throughput measurements still have to prove it.

Replace mmBERT with an encoder built for Ark

mmBERT gave us the multilingual base and shared representation we needed to prove the architecture. It was not designed specifically for this runtime.

Our longer-term goal is to replace it with an encoder built around Ark's constrained decision space, tasks, and execution patterns. We want to reduce both model size and runtime memory again, with a target of roughly 200 to 300 MB for the complete stack. At that point Ark moves beyond normal laptops and servers toward gateways, appliances, and other IoT-class deployments.

The work is part of Benedikt Veith's bachelor thesis. The thesis will publish the exact architecture, measurements, and ablations in more detail. We wanted to introduce the direction earlier because the systems conclusion is already becoming clear.

The constraints turned into an advantage

We did not choose the easy infrastructure for Ark. We chose the machines the security layer eventually had to protect: ordinary endpoints, including older 8 GB laptops without a GPU.

That forced us to build German data, quantize aggressively, unify the classifiers, share the representation, and stop treating every input as if it needed the full model. The result is not finished, but it now runs continuously in our Desktop application on an ordinary M1 laptop.

More importantly, the architecture no longer looks like a compromise made for weak hardware. Selective execution, shared representations, predictable local costs, and keeping sensitive data on the device are useful properties on a server too. The endpoint constraint may turn out to be the reason Ark scales better, not the reason it scales worse.

In Part II, we will open the NTDB itself: how its classifiers work together, how attention combines evidence across chunks, and how a small Layer-2 architecture learns when its own decision is sufficient.

If you are working on efficient inference, early-exit architectures, AI-agent security, or on-device ML, we would be happy to compare notes.