KG Extraction — KGFromDocument Pipeline

KGFromDocument.py extracts a Knowledge Graph from a plain-text or HTML document using an ontology defined in Neo4j. The output is a set of Entity and Relationship nodes persisted in the selected permanent Community ID.

Communities are dual-labeled as :Community:Dataset. This allows a Community to be handled by Dataset-oriented services and visualizations while preserving its role as the ontology-scoped namespace for persistent knowledge graph entities. The relationship is one-way: a Dataset is not necessarily a Community; most Datasets are filtered or materialized result sets without Community semantics.

Community IDs are bare, permanent Dataset IDs and are not session-scoped. This is the same permanence convention used by Dataset services for any permanent Dataset; the Community label is what distinguishes ontology-defined knowledge graph namespaces from other permanent Datasets. A Community has no separate name property: its id is the canonical value used for lookup, navigation, membership, and rename operations.

Multiple Communities per Ontology

An ontology describes the structure and vocabulary used to extract a knowledge graph. It may define any number of communities, including none, one, or several. A community is the ID-addressed namespace in which the entities and relationships extracted under that ontology are stored. Communities are therefore siblings under an ontology, not mutually exclusive alternatives.

A document follows an ontology through (:Document)-[:FOLLOWS]->(:Ontology). Because one ontology can define several communities, that document is related to the complete set of communities defined by the ontology. The document-to-ontology relationship is singular in this model; the ontology-to-community relationship is one-to-many.

Entity membership is represented independently through (:Community)-[:AGGREGATES]->(:Entity). An entity may be aggregated by zero, one, or multiple communities. Multiple memberships do not imply duplicate entities: they indicate that the same knowledge-graph entity is visible in more than one ontology-defined namespace.

Community IDs are permanent identifiers. They should be treated as a collection whenever a document, entity, search request, or UI route can refer to more than one community. A scalar community value can only be a compatibility or display choice and must not be interpreted as the complete membership of the resource.

Query cardinality

Queries that return a document or entity together with related communities must preserve one logical row for the primary resource. Matching a community as a scalar creates one row per community and can make a single document appear several times. Aggregate the related nodes instead:

OPTIONAL MATCH (document)-[:FOLLOWS]->(ontology:Ontology)
OPTIONAL MATCH (ontology)-[:DEFINES]->(community:Community)
WITH document, conf, ontology, collect(DISTINCT community) AS communities
RETURN document, conf, ontology, communities

The consumer reads communities as a collection and removes duplicate IDs while preserving the complete set. A final DISTINCT on a projection that still contains scalar community is not sufficient: rows with different communities are legitimately distinct and will remain separate.

The same rule applies to entity lookups. When communities are discovered through (:Community)-[:AGGREGATES]->(:Entity), return distinct communities and expose all matching IDs. Access control is evaluated for each community, so an inaccessible community is omitted without discarding other accessible memberships.

Ontology Attribute Datatypes

Runtime nodes use labels such as :Entity:Animal and relationships use types such as :is_friend_of. Their authoritative attribute definitions are stored separately on :EntityOntology:Animal and :RelationshipOntology:is_friend_of nodes under the same :Ontology node.

Each ontology attribute stores its datatype in a leading type hint, for example (int) Age in years, (float) Confidence, (bool) Is internal, or (string) Name. The Blazor ontology editor reads and writes this format. Consolidation must use the matching ontology definition when converting values: a single source value does not need an LLM call, but an int attribute should be converted deterministically to an integer while a string attribute must remain text. If conversion fails, the original value is retained as text. (date) attributes are intentionally retained as strings for now. TODO: convert declared dates to native Neo4j date values once the write path supports Neo4j temporal values consistently.

Every runtime Entity and Relationship belongs to exactly one ontology. The consolidation service relies on that invariant when resolving the ontology definition for an entity label or relationship type.

Consolidated values are currently interpolated into the generated Cypher after basic string escaping. Future improvement: use parameterized Cypher map assignments for consolidated entity and relationship properties.

Inputs

Input Description
document_text HTML string of the document to analyse. Paragraph/section elements must carry id attributes for provenance tracking (e.g. <p id="para-3">).
ontology_id Neo4j id of the :Ontology node that drives extraction. Defines which entity and relationship types to look for, their descriptions, and their attributes.
community_id Neo4j id of the target permanent :Community:Dataset for this extraction run. An ontology may define multiple communities; each run writes to its selected target namespace, while readers and consumers preserve the full set of communities associated with a document or entity.
document_id Neo4j id of the source document; stored on each persisted node as document_or_chapter_id.
stop_after KGExtractionPhase enum value that controls how far the pipeline runs (useful for debugging or incremental extraction).

Completion and status reporting

KgFromDocument reports status through both the standard skill return value and the shared live progress channel:

Progress is updated at setup, after community creation, after entity extraction, and after relationship extraction, with additional updates per entity type and relationship type. Early input failures return before the first progress assignment, so callers must use the return dictionary to distinguish failure from an incomplete or stale progress value. The general contract is documented in SKILLS.md.

Document Splitting

Two helpers produce different views of the document:

Helper Output Used by
_chunk_document() Groups consecutive top-level HTML elements into text blocks of ≤ CHUNK_MAX_CHARS (3000) characters. Retains id attributes so the LLM can emit element-level provenance from within a chunk. All phases

_chunk_document() algorithm

strip_html_element_attributes is applied to each element before size measurement — all HTML attributes except id are stripped, so element ids survive into the chunk (the LLM uses them to emit source_id).

1. Parse the HTML and collect top-level elements.
2. For each top-level element:
   a. If len(element_html) ≤ CHUNK_MAX_CHARS  →  add to flat list as-is.
   b. Else if element has child elements (container)  →  expand one level:
         replace the container with its direct child elements
         (each child added individually to the flat list).
   c. Else (oversized leaf with no element children)  →  add as-is regardless of size
         (no further split is possible without breaking content structure).
3. Accumulate the flat list into character-budget chunks:
   - Start a new chunk whenever adding the next element would exceed CHUNK_MAX_CHARS.
   - Join elements within a chunk with '\n'.
4. Fall back to [document_text] as a single chunk if HTML parsing fails or yields no elements.

A single LLM call therefore never receives more than ~3000 characters of HTML, except in the leaf-overflow edge case (step 2c).

LLM Input Format

Every LLM call is a multi-message conversation assembled from the following slots:

# Role Content Present when
1 system Task description, output YAML schema, RULES, and EXAMPLES Always
2 system Existing entities already in the community (Phase 1) or already-known attribute values (Phase 2) Only when the community already contains relevant data for the entity type / attribute being extracted
3 system <DOCUMENT_CONTEXT> — the whole document as plain text (HTML stripped via generate_text_from_html()). Falls back to the document's stored summary if the plain text exceeds MAX_LOADED_DOCUMENT_LENGTH_DEFAULT. Absent if both are empty (large document with no summary). Present for most documents; absent only when len(plain_text) > MAX_LOADED_DOCUMENT_LENGTH_DEFAULT and the document has no stored summary
4 user <TEXT_BLOCK_IN_FOCUS> — the chunk (Phases 1, 3, 4) or single element (Phase 2) as HTML with id attributes preserved (all other HTML attributes stripped). This is the only source the LLM is allowed to extract from. Always

The same document_context string (slot 3) is reused unchanged across all LLM calls for a given document run.

Example user message (slot 4):

<TEXT_BLOCK_IN_FOCUS>
<p id="para-3">Charly, a big fluffy dog with a brown coat, wagged his tail.</p>
<p id="para-4">Tom handed the letter to Sarah and sat down.</p>
</TEXT_BLOCK_IN_FOCUS>

The LLM is instructed to copy the id attribute of the most specific HTML element containing the evidence into the source_id field of its output.

Extraction Phases

Phase Name LLM function Granularity Output Notes
1 Entity Discovery _call_llm_extract_entities() entity type × chunk [{entity_id, name}] Name variants collected across chunks and collapsed into {entity_id, name, names[]} before persistence.
1b Entity Deduplication _call_llm_deduplicate_entities() entity type (no document scan) {dedup_map, elements_to_remove} Aliases merged under canonical id; wrong-type entries dropped.
2 Entity Attribute Extraction _call_llm_extract_entity_attribute() entity type × attribute × chunk [{entity_id, value, source_id}] LLM emits source_id pointing to the specific element id within the chunk. All calls for a type dispatched in parallel.
3 Relationship Discovery _call_llm_extract_relationships() relationship type × chunk [{source_entity_id, target_entity_id, source_id}] LLM emits source_id from inside the chunk. Duplicate (source, target) pairs deduplicated in Python; pairs with unknown entity ids discarded.
4 Relationship Attribute Extraction _call_llm_extract_relationship_attribute() relationship type × attribute × chunk [{source_entity_id, target_entity_id, value}] Chunk granularity sufficient — relationship identity is fixed by Phase 3; sub-element provenance not needed.

LLM Call Count Formula

Total calls =
  (E_types × Chunks)                         -- Phase 1: entity discovery
  + E_types                                  -- Phase 1b: deduplication
  + (sum over E_types of Attrs × Chunks)     -- Phase 2: entity attributes
  + (R_types × Chunks)                       -- Phase 3: relationship discovery
  + (sum over R_types of Attrs × Chunks)     -- Phase 4: relationship attributes

Phase 2 can still dominate for ontologies with many attributes and many entity types, but the call count is now bounded by chunks (not elements).

Example — bob-01.txt with FictionalStories ontology

Variable Value
Entity types (E_types) 4
Total entity attributes 7
Relationship types (R_types) 3
Total relationship attributes 0
Chunks ~4

$$\underbrace{4 \times 4}{P1} + \underbrace{4} + \underbrace{7 \times 4}{P2} + \underbrace{3 \times 4} + \underbrace{0}_{P4} = 16 + 4 + 28 + 12 \approx 60 \text{ calls}$$

Previously ~123 calls (Phase 2 was element-based: 7 × 13 = 91). Now ~60 — a ~50% reduction.

Provenance (source_id)

Every persisted attribute value or relationship carries a paragraph_id in Neo4j that traces back to the HTML element where the evidence was found:

Partial Runs (KGExtractionPhase)

The stop_after parameter accepts a KGExtractionPhase enum value:

Value Runs through
ENTITIES Phase 1 + 1b only
ENTITY_ATTRIBUTES Phases 1 – 2
RELATIONSHIPS Phases 1 – 3
RELATIONSHIP_ATTRIBUTES Full run (default)

Output (Neo4j)

All results are written via KnowledgeGraphEditor:

Helper Creates / updates
entity_create_or_update() :Entity:<label> node; AGGREGATES edge from :Community; PART_OF edge from :Document
relationship_create_or_update() Typed relationship between two :Entity nodes with optional paragraph_id