Thalabus Verification Architecture
Overview
Thalabus employs a several layers of verification architecture to ensure high-quality, stable, and correct agentic outputs. Two independent systems operate at different stages of the generation pipeline, while a third layer (Knowledge Graph Ingestion System) ensures consistency across multiple sources. They are:
- de²generator — a real-time streaming verifier that monitors token-level generation and prevents degeneration, ensuring the answer remains stable and coherent as it is produced.
- AgentVerifier — an asynchronous post-answer evaluator that analyzes the fully completed answer and produces a detailed correctness report accessible via the GUI.
- Knowledge Graph Conflict Management — a system that ensures consistency and reliability of information across multiple sources, supporting both real-time and asynchronous verification layers.
Together, they form a robust multi-layer verification pipeline suitable for high-assurance agentic systems. This document describes their roles, differences, and evaluation criteria.
Verification Layers
1. Real-Time Streaming Verification (de²generator)
de²generator operates during token generation. Its responsibilities include:
- Detecting and preventing semantic drift
- Blocking repetition loops
- Validating syntax and structural integrity
- Recording valid and suspect checkpoints
- Restarting generation from last valid checkpoint when degeneration occurs
- Providing optional real-time translation
- Ensuring verification is faster than the LLM token emission rate
de²generator is designed for low-latency, real-time operation, ensuring that the agentic answer remains coherent and stable as it is generated. de²generator can be monitored in real-time via its own GUI (admin access only, can be disabled for security). de²generator is independent on thalabus and on LLMs, but it is integrated with the thalabus tooling ecosystem and acts as a proxy between thalabus and the (local/remote) LLMs. It can be chained to traditional LLM gateways, load balancers etc.
2. Asynchronous Post-Answer Verification (AgentVerifier)
AgentVerifier operates after the agentic answer is fully generated. Its responsibilities include:
- Full semantic correctness evaluation
- Context fidelity analysis
- Structural coherence assessment
- Stylistic alignment verification
- Safety and hallucination detection
- Producing a detailed verification report
- Displaying a green (OK) or red (Problems) icon in the GUI for each generated answer
- Allowing users to click the icon to open the full report
AgentVerifier has no real-time constraints and may perform deep, multi-pass analysis. AgentVerifier works asynchronously, without blocking the conversation flow. Its results are typically available within a few seconds after the answer is generated.
3. Knowledge Graph Conflict Management
Not described in this document.
Real-Time vs Asynchronous Verification
| Dimension | de²generator (Real-Time Streaming) | AgentVerifier (Asynchronous Post-Answer) |
|---|---|---|
| Architecture Role | Guardian of generation | Guardian of correctness, avoider of hallucinations |
| Timing | During token generation | After full answer completion |
| Primary Goal | Prevent degeneration and maintain stability | Evaluate correctness, fidelity, structure, style, safety |
| Performance Constraint | Must be faster than LLM token emission | No real-time constraints |
| Input Visibility | Sees tokens incrementally | Sees entire final answer |
| Context Awareness | Partial, evolving | Full context + full answer |
| Evaluation Axes | Degeneration, drift, repetition, syntax validity, checkpoint stability | Semantic correctness, context fidelity, structural coherence, stylistic alignment, safety |
| Actions | Block, restart, correct, translate | Score, annotate, flag, classify |
| Output | Stable streaming answer | Verification icon + full report |
| Translation | Real-time experimental translation | Full translation evaluation possible |
| Error Handling | Instant recovery via checkpoint restart | Deep diagnostics allowed |
| User Experience | Smooth, stable streaming | Post-hoc quality indicator |
| Failure Mode | Token-level degeneration | Incorrectness, hallucination, misalignment |
| Scope | Micro-semantic, token-level | Macro-semantic, answer-level |
Evaluation Axes Coverage by Verification Layer
Thalabus evaluates textual output along five independent axes during both real-time and asynchronous verification. The following table defines each axis, breaks down its components, and describes how each verification layer addresses it.
| Evaluation Axis | Component | de²generator (Real-Time Streaming) | AgentVerifier (Asynchronous Post-Answer) | Knowledge Graph Integrity | Roadmap |
|---|---|---|---|---|---|
| Semantic Correctness Evaluates whether the answer is true, logically consistent, and factually grounded. |
n/a | Partially Implemented | |||
| Factual grounding | n/a | Evaluates factual grounding against context | n/a | Implemented | |
| Logical consistency | n/a | Full logical consistency evaluation | n/a | On Request | |
| Causal coherence | n/a | Evaluates causal coherence in completed answer | n/a | On Request | |
| Inference validity | n/a | Full inference validity evaluation | n/a | On Request | |
| Context Fidelity Evaluates whether the answer respects and uses the provided context; displays the referenced context for user inspection. |
n/a | Partially Implemented | |||
| Context visualization | n/a | Provides identification and visualization of referenced context down to page/chapter/paragraph level | n/a | Implemented | |
| Context obedience | n/a | Full context obedience evaluation | n/a | Implemented | |
| Relevance density | n/a | Evaluates relevance density of final answer | n/a | On Request | |
| Scope discipline | n/a | Evaluates scope adherence in final answer | n/a | On Request | |
| Structural Coherence Evaluates the organization and flow of the answer. |
n/a | Soon | |||
| Narrative coherence | n/a | Evaluates narrative coherence of complete answer | n/a | Soon | |
| Information ordering | n/a | Evaluates overall information ordering | n/a | Soon | |
| Completeness vs conciseness | n/a | Evaluates balance in completed answer | n/a | Soon | |
| Redundancy detection | n/a | Detects redundancy in final answer | n/a | Soon | |
| Stylistic Alignment Evaluates tone, persona, and formatting compliance. |
n/a | On Request | |||
| Tone alignment | n/a | Evaluates tone consistency throughout | n/a | On Request | |
| Audience fit | n/a | Evaluates audience fit of final answer | n/a | On Request | |
| Instruction compliance | n/a | Evaluates compliance with all instructions | n/a | On Request | |
| Persona consistency | n/a | Evaluates persona consistency in final answer | n/a | On Request | |
| Degeneration Risk and Correction Evaluates whether the answer exhibits signs of collapse or drift. If detected, take immediate corrective action via checkpoint restart without waiting for full answer generation. |
n/a | Implemented | |||
| Semantic drift | Detects and corrects via checkpoint restart | n/a | n/a | Implemented | |
| Mode collapse | Detects and corrects via checkpoint restart | n/a | n/a | Implemented | |
| Repetitive phrasing | Detects and corrects via checkpoint restart | n/a | n/a | Implemented | |
| Token-level instability | Detects and corrects via checkpoint restart | n/a | n/a | Implemented | |
| Token budget exhaustion (over-thinking) | Detects and corrects via checkpoint restart | n/a | n/a | Implemented | |
| Syntactical Risk and Correction Evaluates whether structured data (JSON, XML, code, etc.) conforms to required syntax, schema, and format rules. |
— | — | — | n/a | Partially Implemented |
| Schema validation | Detects schema violations during generation and corrects via checkpoint restart | n/a | n/a | Soon | |
| Bracket/tag matching | Detects and corrects mismatched brackets, parentheses, and tags in real time | n/a | n/a | Implemented | |
| Quote escaping | Detects and corrects quote escaping errors during generation | n/a | n/a | Implemented | |
| Indentation consistency | Detects indentation errors during generation | n/a | n/a | On Request | |
| Data type correctness | Detects type mismatches during generation | n/a | n/a | On Request | |
| Cross-Source Consistency ⚠️ Evaluates whether multiple information sources contain conflicting or inconsistent information during ingestion and knowledge graph creation. Out of scope for this paper. This is a part of the Knowledge Graph Ingestion System. |
— | n/a | n/a | Detects contradictions across documents stored in the graph; supports governance quality processes. | Soon |
| Conflict detection | n/a | n/a | Identifies conflicting facts from different sources comparing node properties | Soon | |
| Ingestion-time inconsistency flagging | n/a | n/a | Flags contradictions during sources ingestion | Soon | |
| Source reliability weighting | n/a | n/a | Weights conflicting facts by source reliability to determine truth | Soon | |
| Knowledge graph contradiction identification | n/a | n/a | Traverses graph relationships to find logical contradictions in stored knowledge | On Request | |
| Identifies redundant sources | n/a | n/a | Detects redundant information in the information sources, enables removal of obsolete documents | On Request | |
| Identifies redundant knowledge | n/a | n/a | Detects redundant information in the knowledge graph, enables consolidation | On Request |