DocumentTextEnhancer Skill

Last Updated: June 23, 2026
Status: ✅ Implemented and tested — Successfully reconstructed docid-8230 from 6 chapters/23 paragraphs to 1 chapter/31 paragraphs

Overview

The DocumentTextEnhancer skill processes raw PDF-extracted text from Document nodes and: 1. Sanitizes and formats the text into clean HTML with semantic structure 2. Stores the enhanced HTML in Document.enhanced_text 3. Reconstructs a single Chapter node with text from the enhanced HTML 4. Extracts Paragraph nodes from HTML elements with id attributes 5. Calculates embeddings for both Chapter and Paragraphs

Data Flow Architecture

Strategy: Document → Chapter → Paragraphs

:Document
  ├── text (original raw PDF text)
  └── enhanced_text (sanitized HTML) ──┐
                                        │
                                        ↓
                                   :Chapter (ONE per document)
                                     ├── text (plain text from enhanced_text)
                                     ├── embedding (calculated from text)
                                     └── enhanced_text (copy of Document.enhanced_text)
                                          │
                                          ↓
                                     :Paragraph nodes (extracted from HTML id elements)
                                       ├── text (extracted with overlap)
                                       └── embedding (calculated from text)

Key Properties

Node Type Property Source Usage
Document text Original PDF extraction Preserved as-is; never modified
Document enhanced_text DocumentTextEnhancer output Source for Chapter reconstruction
Chapter text Plain text from Document.enhanced_text Used for embeddings and fulltext search
Chapter enhanced_text Copy of Document.enhanced_text Preserved for reference
Paragraph text Extracted from HTML elements (with overlap) Used for embeddings and vector search

HTML Structure

Input: Raw PDF Text

Output: Enhanced HTML

Example Enhanced HTML

<p id="text-001" class="document-header">
Case Number: D-1234/2024
Court: Federal Administrative Court
</p>

<p id="text-002">
The appellant argues that the decision was based on insufficient evidence...
</p>

<p>
<span id="text-003-1">First part of a long paragraph that exceeds the threshold.</span>
<span id="text-003-2">Second part of the same paragraph continues here.</span>
</p>

Text Sanitization Pipeline

Phase 1: Cleanup (_sanitize_pdf_text)

  1. Remove all HTML tags (keep text content)
  2. Remove <br/> tags and surrounding whitespace
  3. Remove page footers/headers (e.g., "Seite 5", "D-1234/2024")
  4. Fix hyphenated words split across lines: word -\n word → wordword
  5. Split by paragraph breaks (\n\n+)
  6. Merge redaction markers (underscore-only lines) with adjacent text
  7. Merge continuation lines (starting with comma or lowercase)
  8. Convert single newlines to spaces
  9. Collapse multiple spaces (except in document header)
  10. Wrap each paragraph in <p id="text-NNN"> tags

OCR Error Detection and Correction

Problem Statement: OCR Data Quality Issues

Many documents in dossier-jurispub suffer from OCR artifacts that degrade data quality:

Example 1 - Spaced Characters: "Arrêtdu 1 2 mars 2 0 2 4" - Individual letters separated by spaces: "A r r ê t", "S e n t e n z a" - Individual digits separated by spaces: "1 2", "2 0 2 4" - Pattern: 3+ alphanumeric chars separated by whitespace

Example 2 - Word Collation: "Sentenzadel", "SentenzaDelCorte" - Multiple words stuck together without spaces - Pattern: Unusually long words (>15 chars) or abnormal CamelCase in Romance language text

Impact: - Poor embedding quality: corrupted text → poor semantic vectors - RAG retrieval failures: misspellings miss relevant documents - Data integrity: case numbers and dates become garbled

OCR Correction Strategy (Option 1: Post-Extraction)

Timing: After paragraph extraction, before embedding calculation

Three new functions detect and fix OCR errors:

  1. _detect_ocr_patterns(text) → Dict[str, List[str]]
  2. Fast heuristic detection (no LLM)
  3. Detects spaced chars: \b[a-zA-Z0-9]\s+(?:[a-zA-Z0-9]\s+){2,}[a-zA-Z0-9]\b
  4. Detects collated words: long words (>15 chars) or abnormal CamelCase

  5. Returns patterns found or empty dict (skip LLM if no patterns)

  6. _detect_and_fix_ocr_errors(para_text, llm_manager) → Optional[str]

  7. Invokes LLM only if OCR patterns detected (efficiency!)
  8. LLM prompt instructs fixing spaced chars and collated words
  9. Returns corrected text or None if no LLM call needed

  10. Error handling: returns None on LLM failure; original text used for embedding

  11. Updated _reconstruct_chapter_and_paragraphs()

  12. New parameter: llm_manager=None
  13. For each paragraph:
    • Detect OCR patterns using heuristics
    • If found, call LLM to fix
    • Update paragraph text with corrected version
    • Update enhanced_text HTML with fixed text (best-effort replacement)
  14. Build chapter text from all fixed paragraph texts
  15. Calculate chapter embedding on fixed text
  16. Calculate paragraph embeddings on fixed texts
  17. Store all corrected texts in Neo4j

Correctness Guarantee: All Embeddings Use Fixed Text

Enhanced HTML → Extract paragraphs → Detect OCR → LLM fix (if needed)
                                                         ↓
                                        Paragraph text FIXED
                                                         ↓
                                        Paragraph embedding (on FIXED text) ✓

All fixed paragraphs → Join texts → Chapter text FIXED
                         ↓
                   Chapter embedding (on FIXED text) ✓

Result: Both paragraph and chapter embeddings improve quality through LLM-corrected text.

Configuration

OCR detection thresholds in DocumentTextEnhancer.py:

OCR_LONG_WORD_THRESHOLD = 15        # Words > 15 chars trigger detection
OCR_SPACED_CHARS_MIN = 3            # 3+ spaced chars trigger detection

Adjust if experiencing false positives/negatives: - Increase OCR_LONG_WORD_THRESHOLD to reduce false positives on legitimate long words - Decrease OCR_SPACED_CHARS_MIN to catch more spaced-char patterns (risk of false positives)

Performance

Phase 2: Long Paragraph Splitting (_split_long_paragraphs)

Paragraph Extraction

HTML Parsing (_extract_paragraphs_from_html)

Example with Overlap

HTML:

<p id="text-001">First paragraph.</p>
<p id="text-002">Second paragraph.</p>
<p id="text-003">Third paragraph.</p>

Extracted Paragraphs (overlap=1): - Paragraph 1: "First paragraph. Second paragraph." (text-001 + overlap) - Paragraph 2: "First paragraph. Second paragraph. Third paragraph." (overlap + text-002 + overlap) - Paragraph 3: "Second paragraph. Third paragraph." (overlap + text-003)

Why Overlap?

Chapter and Paragraph Reconstruction

Workflow (_reconstruct_chapter_and_paragraphs)

  1. Create Chapter ID: {docid}-chapter-001 (single chapter per document)
  2. Extract Plain Text: Strip HTML tags from enhanced_text for embedding calculation
  3. Calculate Chapter Embedding: Use EmbeddingsManager on plain text
  4. Store Chapter Data:
  5. text: Plain text version
  6. title: "Chapter 1"
  7. embedding: Vector embedding
  8. enhanced_text: Full HTML (copy of Document.enhanced_text)
  9. enhanced_type: "html"
  10. Extract Paragraphs: Parse HTML for id elements with overlap
  11. Calculate Paragraph Embeddings: For each extracted paragraph
  12. Update Neo4j:
  13. Delete existing Chapter and Paragraph nodes (to avoid duplicates)
  14. Create/update single Chapter node
  15. Batch create/update all Paragraph nodes

Important: One Chapter Per Document

Embeddings

Calculation

Chapter Embedding

Paragraph Embeddings

Neo4j Indexes

No Changes to Index Structure

DocumentTextEnhancer uses the existing single-index strategy:

Index Name Node Type Property Purpose
chapters_fulltext Chapter text Fulltext search (case-sensitive/insensitive)
typical_rag Chapter embedding Vector semantic search
chapterparagraph Paragraph embedding Vector semantic search

No dual-index complexity — all indexes operate on text property, which now contains enhanced content.

Migration Strategy

From Original RagIngestion to DocumentTextEnhancer

Before: - Document → Multiple Chapters (token-based chunking) - Chapters → Paragraphs (smaller token-based chunks)

After DocumentTextEnhancer: - Document → ONE Chapter (entire document as single chapter) - Chapter → Paragraphs (extracted from HTML id elements with overlap)

Data Migration Process

  1. Run DocumentTextEnhancer on document: python from src.core.extensions.skillsets.Documents.DocumentTextEnhancer import DocumentTextEnhancer result = DocumentTextEnhancer(docid="docid-8230", read_only=False)

  2. What happens:

  3. Document.enhanced_text is created/updated
  4. Existing Chapter/Paragraph nodes are deleted
  5. New single Chapter node is created
  6. New Paragraph nodes are extracted and created

  7. All embeddings are recalculated

  8. Rollback (if needed):

  9. Delete Chapter and Paragraph nodes
  10. Re-run RagIngestion on the document to restore original structure

Testing Checklist

✅ Verify Document.enhanced_text contains valid HTML
✅ Verify single Chapter node exists with correct text and embedding
✅ Verify Paragraph nodes extracted match HTML id elements
✅ Verify paragraph text includes overlap context
✅ Verify all embeddings are non-empty vectors
✅ Test RAG queries return relevant fragments from enhanced content
✅ Compare RAG quality before/after enhancement

Usage

Command-Line Testing

# Run with read_only=True (no Neo4j update)
python src\core\extensions\skillsets\Documents\DocumentTextEnhancer.py

# Or import and call directly
from src.core.extensions.skillsets.Documents.DocumentTextEnhancer import DocumentTextEnhancer

# Read-only mode (debugging, output files only)
result = DocumentTextEnhancer(
    docid="docid-8230",
    read_only=True,
    output_filename_base="docid-8230"
)

# Production mode (updates Neo4j)
result = DocumentTextEnhancer(
    docid="docid-8230",
    read_only=False
)

Return Structure

{
    "docid": "docid-8230",
    "status": "completed",
    "message": "Enhancement completed",
    "original_length": 12345,
    "enhanced_length": 15678,
    "read_only": False,
    "enhanced_text": "<p id=\"text-001\">...</p>...",
    "reconstruction": {
        "status": "completed",
        "chapter_id": "docid-8230-chapter-001",
        "paragraph_count": 42
    }
}

Test Results: docid-8230 (June 23, 2026)

Successful validation with real document:

Before DocumentTextEnhancer

After DocumentTextEnhancer

Key Observations

✅ All original Chapter/Paragraph nodes successfully deleted
✅ Single Chapter node created with correct structure
✅ 31 Paragraphs extracted from HTML <p id="text-NNN"> elements
✅ Embeddings calculated for Chapter and all Paragraphs
✅ Enhanced text properly formatted with overlapping context
✅ No data loss; all content preserved in new structure

Query Verification

Used script: PythonServer/scripts/query_document_hierarchy.py

python scripts/query_document_hierarchy.py docid-8230

Output:

SUMMARY: 1 Document → 1 Chapter(s) → 31 Paragraph(s)

Configuration

Paragraph Splitting Thresholds

# In DocumentTextEnhancer.py
LONG_PARAGRAPH_THRESHOLD = 500  # Target size for each span
LONG_PARAGRAPH_SPLIT_THRESHOLD = 1000  # Only split if exceeds 2×
FINE_CUT_CHARACTERS = set('.;:!?,-()[]{}"\'/')  # Good cut points
FINE_CUT_SEARCH_RANGE = 50  # Search range for fine cuts

Paragraph Overlap

# In _extract_paragraphs_from_html()
overlap_elements = 1  # Number of surrounding elements to include

Embedding Model

# Configured in EmbeddingsManager (via environment or config)
# Default: OpenAI text-embedding-3-large, 3072 dimensions

Known Limitations

  1. One Chapter Per Document: Cannot preserve multi-chapter structure from original RagIngestion
  2. Overlap Duplication: Paragraphs include overlapping content, increasing storage/embedding costs
  3. HTML Dependency: Requires well-formed HTML with id attributes; if HTML is malformed, paragraph extraction may fail
  4. Destructive Update: Deletes existing Chapter/Paragraph nodes before reconstruction (no merge logic)
  5. No Incremental Updates: Must reprocess entire document even if only small changes needed

Future Enhancements

Related Documentation