Neo4j Indexes and Embeddings Reference

Last Updated: June 23, 2026
Status: ✅ Validated — Single-index strategy implemented and tested with docid-8230

This document provides a comprehensive inventory of all Neo4j indexes in the Thalabus system, including vector indexes for semantic search, fulltext indexes for text search, and range indexes for lookups. It also clarifies the DocumentTextEnhancer strategy for populating text properties from enhanced_text.

Quick Reference: Search Capabilities by Node Type

Node Type	Vector Index	Fulltext Index	Range Indexes	Embeddings Property	Source
`Chapter`	✅ `typical_rag`	✅ `chapters_fulltext`	✅ `id`, `documentid`	`text` (plain text from enhanced_text)	Document.enhanced_text
`Paragraph`	✅ `chapterparagraph`	❌ None	✅ `id`, `documentid`	`text` (extracted from HTML)	Chapter HTML elements
`Document`	❌ None	❌ None	✅ `id`, `documentid`	N/A (not embedded)	N/A
`Summary`	✅ `summary`	❌ None	✅ `id`, `documentid`	`text` (raw summary text)	RagIngestion
`Keyword`	✅ `keywords`	✅ `keywordsTextIndex`	✅ `id`	`text` (raw keyword text)	RagIngestion
`Question`	✅ `hypothetical_questions`	❌ None	✅ `id`	`text` (raw question text)	RagIngestion
`LlmHint`	✅ `hints`	❌ None	✅ `id`	`text` (raw hint text)	AgentHints
`Entity`	✅ `ontology_entity`	❌ None	✅ `id`	`text` (raw entity text)	Ontology

DocumentTextEnhancer Strategy

Key Architectural Decision: Document.enhanced_text serves as the source for Chapter and Paragraph text properties.

Data Flow

:Document
  ├── text (original raw PDF text - preserved)
  └── enhanced_text (sanitized HTML) ──────┐
                                            │
                                            ↓
                                       :Chapter (ONE per document)
                                         ├── text (plain text from enhanced_text)
                                         ├── embedding (from text)
                                         └── enhanced_text (copy of Document.enhanced_text)
                                              │
                                              ↓
                                         :Paragraph nodes (extracted from HTML)
                                           ├── text (HTML elements with overlap)
                                           └── embedding (from text)

Property Usage

Property	Node	Source	Used For
`text`	Document	PDF extraction	Preserved as-is; never modified
`enhanced_text`	Document	DocumentTextEnhancer	Source for Chapter reconstruction
`text`	Chapter	Plain text from Document.enhanced_text	✅ Embeddings ✅ Fulltext search
`enhanced_text`	Chapter	Copy of Document.enhanced_text	Preserved for reference
`text`	Paragraph	Extracted from HTML elements	✅ Embeddings ✅ Vector search

Migration Path

Original RagIngestion: Document → Multiple Chapters (token-based) → Paragraphs
After DocumentTextEnhancer: Document → ONE Chapter (entire document) → Paragraphs (HTML-based with overlap)

See DOCUMENTTEXTENHANCER.md for complete implementation details.

Vector Indexes (Semantic Search)

All vector indexes enable semantic similarity search via embedding vectors. Embeddings are calculated from the text property using OpenAI's text-embedding-3-large model with configurable dimensions (default: 3072).

The vector index property in Neo4j is always called embedding (this is the stored vector, not the source property).

Index Name	Node Type	Source Property	Vector Property	Dimensions	Similarity	RAG Use
`typical_rag`	`Chapter`	`text`	`embedding`	3072	cosine	✅ Primary
`chapterparagraph`	`Paragraph`	`text`	`embedding`	3072	cosine	✅ Primary
`summary`	`Summary`	`text`	`embedding`	3072	cosine	✅ Fallback
`keywords`	`Keyword`	`text`	`embedding`	3072	cosine	✅ Fallback
`hypothetical_questions`	`Question`	`text`	`embedding`	3072	cosine	✅ Fallback
`ontology_entity`	`Entity`	`text`	`embedding`	3072	cosine	❌ Category-only
`hints`	`LlmHint`	`text`	`embedding`	3072	cosine	❌ Agent hints

Key Point: Embeddings are calculated from the text property. For Chapter and Paragraph nodes processed by DocumentTextEnhancer, text contains enhanced content (plain text or HTML-extracted text), so embeddings automatically benefit from the sanitization and enrichment.

Vector Index Creation

Vector indexes are created in src/core/base/neo4j_indexes.py at these points:

Startup: parameters._create_vector_indexes() → ensure_all_indexes()
Document ingestion: documents_service.create_all_required_indexes() → ensure_all_indexes()
Coins ingestion: CoinIngester.prepare_neo4j_database() → ensure_all_indexes()
Lazy (hints): hints_service.create_hints_vector_indexes() (on first hint write)
Lazy (ontology): ontology_service.create_semantic_search_index() (after embedding updates)

Embedding Calculation

For Chapter, Paragraph, Summary, Keyword, and Question nodes:

Source: The text property (plain text)
Model: OpenAI text-embedding-3-large (configurable)
Dimensions: Configurable via EmbeddingsManager (default: 3072)
Storage: Calculated vector stored in embedding property
Execution Triggers:
RagIngestion.py: Original ingestion (lines 183 for chapters, 209 for paragraphs)
DocumentTextEnhancer.py: After reconstructing Chapter/Paragraphs from enhanced_text

Fulltext Indexes (Text Search)

Fulltext indexes enable case-sensitive and case-insensitive text searches via Lucene-based Neo4j fulltext search.

Index Name	Node Type	Property	Consistency	Query Type
`chapters_fulltext`	`Chapter`	`text`	Eventually consistent	Case-insensitive (default) or case-sensitive (`@'...'`)
`keywordsTextIndex`	`Keyword`	`text`	Eventually consistent	Case-insensitive (default) or case-sensitive (`@'...'`)

Fulltext Index Configuration

Consistency: Eventually consistent (default Neo4j configuration)
Query syntax in Thalabus:
Single quotes: 'search text' (case-insensitive)
At-prefix: @'search text' (case-sensitive)

When Fulltext Search is Used

Short semantic queries (≤5 words): Combined with vector search
Explicit text search: Single-quoted queries in the Thalabus query language
Keyword fallback: When semantic search has no results

Range Indexes (Lookups)

Range indexes optimize equality, comparison, and sorting operations. The authoritative list is RANGE_INDEXES in src/core/base/neo4j_indexes.py — the table below was regenerated directly from that source list (previous versions of this table were incomplete/stale: several labels and properties existed in code but were undocumented, and a Question.id range index was listed here that does not actually exist in RANGE_INDEXES).

Node Types with Range Indexes

Node Label	Indexed Properties
`CacheEntry`	`hash`, `guid`, `guid_last_exec`
`Capability`	`id`
`Category`	`text`, `id`
`Chapter`	`id`, `documentid`, `partition`
`Community`	`id`, `isProtected`
`Dataset`	`id`
`Document`	`id`, `documentid`, `isDeleted`, `isProtected`, `partition`
`DocumentTemplate`	`id`, `templateid`
`Dossier`	`id`, `isProtected`
`Entity`	`id`, `is_computed_view`, `is_diagram`, `is_relationship`, `is_structure`
`Group`	`id`
`Image`	`id`
`KeywordCategory`	`text_en`, `id`
`LlmHint`	`key`, `prompt`
`Ontology`	`id`, `isProtected`
`Paragraph`	`id`, `documentid`, `partition`
`ReasoningDocument`	`id`
`Session`	`id`
`Summary`	`id`, `documentid`, `partition`
`TranslationCache`	`key`
`User`	`id`

Note: Keyword and Question node types do not currently have range indexes defined in RANGE_INDEXES, despite both having vector indexes (see the Vector Indexes table above). Ontology, Community, and Dossier each have an isProtected range index supporting ACL protected-resource filtering.

UNIQUE Constraint

Property: User.username
Enforced: Yes
Auto-indexed: Yes (all UNIQUE constraints are automatically indexed)

Index Maintenance

All Indexes Defined In

Single source of truth: src/core/base/neo4j_indexes.py

This module defines: - VECTOR_INDEXES — 7 vector indexes for semantic search - FULLTEXT_INDEXES — 2 fulltext indexes for text search - RANGE_INDEXES — 20 node labels, 44 individual range indexes

Creating Indexes Idempotently

The function ensure_all_indexes(dimension: int = 3072) is safe to call multiple times. It: 1. Checks if each index already exists 2. Creates only missing indexes 3. Returns immediately if all indexes are present 4. Can be called at startup, during maintenance, or during data ingestion

from src.core.base.neo4j_indexes import ensure_all_indexes
ensure_all_indexes(dimension=3072)  # Creates all missing indexes

Index Creation Performance

Vector indexes: Require dimension configuration; can be resource-intensive on large datasets
Fulltext indexes: Fast creation; updates are eventually consistent
Range indexes: Fast creation and updates

Enhanced Text vs. Raw Text

Strategy with DocumentTextEnhancer:

Property	Node	Purpose	Embeddings	Search	Indexing
`text`	Document	Original PDF-extracted content	N/A (not embedded)	N/A	Range index on id
`enhanced_text`	Document	Sanitized HTML from DocumentTextEnhancer	N/A	N/A	N/A
`text`	Chapter	Plain text extracted from Document.enhanced_text	✅ YES	✅ YES (fulltext)	`chapters_fulltext`
`enhanced_text`	Chapter	Full HTML copy from Document.enhanced_text	N/A	N/A	N/A (preserved for reference)
`text`	Paragraph	Extracted from HTML elements with overlap	✅ YES	N/A	`chapterparagraph` vector

Key Insight: - enhanced_text is an intermediate processing step in DocumentTextEnhancer - Once stored in Chapter.text and Paragraph.text, embeddings and fulltext search use the text property - This ensures consistent, high-quality content throughout the system without dual-index complexity

Query Language Mapping

Query Syntax	Search Type	Index Used	Returns
`"phrase"`	Semantic + text (if ≤5 words)	`typical_rag`, `chapterparagraph`, `chapters_fulltext`	Fragments with vector scores
`'text'`	Fulltext (case-insensitive)	`chapters_fulltext`, `keywordsTextIndex`	Fragments with fulltext scores
`@'text'`	Fulltext (case-sensitive)	`chapters_fulltext`, `keywordsTextIndex`	Fragments with fulltext scores
`CATEGORY=VALUE`	Ontology lookup	Range indexes on `Category`, `Entity`	Category fragments

Frequently Asked Questions

Q: Are embeddings calculated on text or enhanced_text?
A: text only. However, with DocumentTextEnhancer, the Chapter/Paragraph text property is populated FROM Document.enhanced_text, so embeddings indirectly benefit from the enrichment. This avoids dual-index complexity while improving content quality.

Q: How does the DocumentTextEnhancer migration work?
A: DocumentTextEnhancer sanitizes Document.text into Document.enhanced_text (HTML), then reconstructs Chapter/Paragraph nodes by extracting plain text from the HTML. This replaces multiple RagIngestion chapters with ONE chapter containing all paragraphs extracted from HTML elements (with overlap for context).

Q: Do I need a dual fulltext index strategy?
A: No. With DocumentTextEnhancer, the single chapters_fulltext index on Chapter.text is sufficient. The text property contains enhanced content by design, so all queries automatically benefit from the enrichment.

Q: Can I do fulltext search on Chapters and Keywords?
A: Yes. Both have fulltext indexes (chapters_fulltext and keywordsTextIndex). Paragraphs do not have a fulltext index (only vector index).

Q: What happens if an index doesn't exist?
A: RAG operations will fall back to slower Neo4j queries. Call ensure_all_indexes() to create missing indexes. It's safe to call on every startup.

Q: Can I search on Document nodes directly?
A: No. Document nodes do not have vector or fulltext indexes. Search operates at the Chapter, Paragraph, Keyword, or Question level, returning fragments. Documents are aggregated from fragments only at the final output stage.

Q: How do I configure embedding dimensions?
A: Pass the dimension parameter to ensure_all_indexes(dimension=3072). Default is 3072 (OpenAI text-embedding-3-large).