title: "Enoki-Encoder reached 69.1% F1 at 0.13 seconds per sentence" slug: "enoki-hallucination-spans-cost-boundary" excerpt: "Enoki links evidence checks to exact answer spans, but its strongest results also show why a detector is only as reliable as its extraction, verifier, and reference context." category: "AI Research" tags:
- "Enoki"
- "hallucination detection"
- "RAG"
- "OpenIE"
- "factuality" author: "ISH Team" read_time: "7"
A hallucination score can warn that an answer contains unsupported material. It may not tell an editor which words to remove or an engineer which factual unit failed. Enoki is an open framework built to preserve that connection.
It extracts relational facts from an answer, checks each one against supplied context, and maps unsupported facts back to their original character spans. In the paper's RAGTruth efficiency comparison, the encoder version reached 69.1% sentence-level F1 at 0.13 seconds per sentence. The authors report that it ran four to ten times faster than competitive baselines.
The design matters as much as the speed claim. Enoki keeps a machine-readable fact connected to the prose a person may need to edit. It also exposes three places where an automated check can break. A missed fact is never verified. An incomplete reference may not support a correct statement. The verifier can assign the wrong label.
One representation does two jobs
Many factuality pipelines split an answer into claims, check them, and run another alignment step to find the corresponding text. Enoki uses text-anchored Open Information Extraction. It creates subject, predicate, and object triples for each sentence while keeping relevant arguments tied to answer spans.
Facts are built incrementally. A coarse statement may be supported, then a later refinement can introduce an unsupported detail. The verifier turns each triple into a hypothesis, compares it with chunks of the reference context, and takes the highest entailment score across those chunks. If the fact is unsupported, Enoki projects the newly added object span into the answer.
The official repository gives a compact example. The context says Apple acquired Beats Electronics in 2014; the answer says 2015. The encoder pipeline returns 2015, character offsets 36 through 40, a structured fact, and an unsupportedness probability. An interface can underline the suspect phrase instead of marking the entire response as defective.
from enoki import EnokiPipeline
detector = EnokiPipeline(
method="encoder",
model="s-nlp/enoki-openie-encoder",
)
result = detector.detect(
context=reference_text,
answer=model_answer,
)
The framework provides three extraction backends. Enoki-LLM uses a CycleOIE-style prompt and an OpenAI-compatible endpoint. Enoki-Encoder uses ModernBERT-large and was trained by distilling incremental triples produced by the LLM extractor. Enoki-Rules applies 35 dependency-parse rules over spaCy and needs no extraction model at runtime. All three feed the same verifier and output format, so a team can trade decomposition quality for latency without rebuilding its review interface.
Enoki is strongest when location matters
On the HalluEntity entity-level benchmark, Enoki-LLM with GPT-OSS-120B reached 55.09 AUPRC. MinIE, the strongest external OpenIE baseline in that table, reached 39.77. Enoki-Rules scored 46.81, Enoki-Encoder scored 44.25, and a GPT-5.2 zero-shot RAGTruth prompt scored 36.63.
Results changed across the span datasets. By Span Coverage F1, Enoki-LLM reached 52.07 on MuSHROOM and 71.15 on PsiloQA, the best displayed results in those columns. It scored 37.32 on RAGTruth, below the 42.20 result from a Qwen3-8B detector fine-tuned directly on RAGTruth. Training on the target distribution still helped.
The lighter backends kept much of the localization performance. Enoki-Rules reached 49.18 on MuSHROOM and 65.73 on PsiloQA. Enoki-Encoder reached 46.96 and 65.51. Its permutation-invariant Hungarian matching objective made a measurable difference. Replacing fixed row-by-row supervision increased encoder scores from 41.02 to 46.96 on MuSHROOM, 27.25 to 34.84 on RAGTruth, and 61.14 to 65.51 on PsiloQA.
The detailed efficiency tables separate the three options. On RAGTruth, Enoki-Encoder and Enoki-Rules each took 0.09 seconds per sentence, compared with 7.88 seconds for Enoki-LLM. On FactCheck-Bench, the totals were 0.13, 0.11, and 11.12 seconds. The paper treats FLOP estimates as approximate comparisons rather than hardware bills. Even with that caveat, extraction accounts for most of the LLM version's latency.
A detector does not establish truth
Enoki compares an answer with the context it receives. If the context is stale, incomplete, or wrong, missing entailment does not prove the answer is false. The EnokiQA dataset card says that its Wikipedia evidence can be incomplete or outdated and that automatic labels may contain extraction, localization, or verification errors.
EnokiQA has 3,990 labeled examples and 19,594 unlabeled examples. Its development and test splits contain 1,995 labeled examples each, balanced across seven generator models. GPT-OSS-120B extracts the triples used for automatic labels, and a Qwen3.5-9B NLI-style verifier checks them. On 100 manually labeled test examples, human-to-human agreement was moderate at the character level, with Cohen's kappa of 0.580. Against adjudicated labels, the automatic pipeline reached 0.867 sentence-level F1 and 0.569 span-level F1.
Colored spans should therefore be review cues, not verdicts. The Enoki encoder model card also says that the model is designed for English text. A team working with another language, or with specialized legal, medical, or financial phrasing, needs a target-domain evaluation set.
Our BAML guide draws the same boundary for structured output: cleaner structure makes downstream handling safer, but it cannot verify the facts inside. Content Credentials make a related distinction for images. Provenance and truth are different checks.
Put the span into a review loop
A retrieval-augmented application should log four artifacts together: the answer span, extracted triple, evidence passage, and verifier score. A reviewer needs to see them together. A highlight without evidence is difficult to audit, while one answer-level score hides the individual judgment.
Thresholds should be calibrated on real traffic from the target domain. The Enoki paper calibrates continuous detectors per dataset when labels exist and falls back to 0.5 for MuSHROOM because it lacks a labeled training subset. That setting can change precision and recall, so it belongs in the audit record.
Extraction coverage and verification accuracy also need separate measurements. Sample responses and ask reviewers whether the extractor found every checkable fact. Then inspect whether the supplied evidence settles each fact. If it does not, return "not established by this context" instead of "false." Agent memory should not become factual evidence, either. Preferences and prior conversation may guide behavior without proving an external claim.
Enoki's interactive demo lets readers inspect the span output. The repository includes evaluation commands for HalluEntity, MuSHROOM, RAGTruth, PsiloQA, FactCheck-Bench, and ANAH. Replaying the evaluation that matches your application is a sounder starting point than adopting the headline number.
Enoki is not a truth machine. It produces a more useful failure report. The 69.1% F1 and 0.13-second result belongs to one evaluation setting. In production, keep every automated judgment attached to the fact, evidence, exact text, and threshold that produced it.
