# KG Extraction — KGFromDocument Pipeline `KGFromDocument.py` extracts a Knowledge Graph from a plain-text or HTML document using an ontology defined in Neo4j. The output is a set of `Entity` and `Relationship` nodes persisted in the selected permanent Community ID. Communities are dual-labeled as `:Community:Dataset`. This allows a Community to be handled by Dataset-oriented services and visualizations while preserving its role as the ontology-scoped namespace for persistent knowledge graph entities. The relationship is one-way: a Dataset is not necessarily a Community; most Datasets are filtered or materialized result sets without Community semantics. Community IDs are bare, permanent Dataset IDs and are not session-scoped. This is the same permanence convention used by Dataset services for any permanent Dataset; the `Community` label is what distinguishes ontology-defined knowledge graph namespaces from other permanent Datasets. A Community has no separate `name` property: its `id` is the canonical value used for lookup, navigation, membership, and rename operations. --- ## Multiple Communities per Ontology An ontology describes the structure and vocabulary used to extract a knowledge graph. It may define any number of communities, including none, one, or several. A community is the ID-addressed namespace in which the entities and relationships extracted under that ontology are stored. Communities are therefore siblings under an ontology, not mutually exclusive alternatives. A document follows an ontology through `(:Document)-[:FOLLOWS]->(:Ontology)`. Because one ontology can define several communities, that document is related to the complete set of communities defined by the ontology. The document-to-ontology relationship is singular in this model; the ontology-to-community relationship is one-to-many. Entity membership is represented independently through `(:Community)-[:AGGREGATES]->(:Entity)`. An entity may be aggregated by zero, one, or multiple communities. Multiple memberships do not imply duplicate entities: they indicate that the same knowledge-graph entity is visible in more than one ontology-defined namespace. Community IDs are permanent identifiers. They should be treated as a collection whenever a document, entity, search request, or UI route can refer to more than one community. A scalar community value can only be a compatibility or display choice and must not be interpreted as the complete membership of the resource. ### Query cardinality Queries that return a document or entity together with related communities must preserve one logical row for the primary resource. Matching a community as a scalar creates one row per community and can make a single document appear several times. Aggregate the related nodes instead: ```cypher OPTIONAL MATCH (document)-[:FOLLOWS]->(ontology:Ontology) OPTIONAL MATCH (ontology)-[:DEFINES]->(community:Community) WITH document, conf, ontology, collect(DISTINCT community) AS communities RETURN document, conf, ontology, communities ``` The consumer reads `communities` as a collection and removes duplicate IDs while preserving the complete set. A final `DISTINCT` on a projection that still contains scalar `community` is not sufficient: rows with different communities are legitimately distinct and will remain separate. The same rule applies to entity lookups. When communities are discovered through `(:Community)-[:AGGREGATES]->(:Entity)`, return distinct communities and expose all matching IDs. Access control is evaluated for each community, so an inaccessible community is omitted without discarding other accessible memberships. --- ## Ontology Attribute Datatypes Runtime nodes use labels such as `:Entity:Animal` and relationships use types such as `:is_friend_of`. Their authoritative attribute definitions are stored separately on `:EntityOntology:Animal` and `:RelationshipOntology:is_friend_of` nodes under the same `:Ontology` node. Each ontology attribute stores its datatype in a leading type hint, for example `(int) Age in years`, `(float) Confidence`, `(bool) Is internal`, or `(string) Name`. The Blazor ontology editor reads and writes this format. Consolidation must use the matching ontology definition when converting values: a single source value does not need an LLM call, but an `int` attribute should be converted deterministically to an integer while a `string` attribute must remain text. If conversion fails, the original value is retained as text. `(date)` attributes are intentionally retained as strings for now. TODO: convert declared dates to native Neo4j date values once the write path supports Neo4j temporal values consistently. Every runtime `Entity` and `Relationship` belongs to exactly one ontology. The consolidation service relies on that invariant when resolving the ontology definition for an entity label or relationship type. Consolidated values are currently interpolated into the generated Cypher after basic string escaping. Future improvement: use parameterized Cypher map assignments for consolidated entity and relationship properties. ## Inputs | Input | Description | |---|---| | `document_text` | HTML string of the document to analyse. Paragraph/section elements must carry `id` attributes for provenance tracking (e.g. `

`). | | `ontology_id` | Neo4j id of the `:Ontology` node that drives extraction. Defines which entity and relationship types to look for, their descriptions, and their attributes. | | `community_id` | Neo4j `id` of the target permanent `:Community:Dataset` for this extraction run. An ontology may define multiple communities; each run writes to its selected target namespace, while readers and consumers preserve the full set of communities associated with a document or entity. | | `document_id` | Neo4j id of the source document; stored on each persisted node as `document_or_chapter_id`. | | `stop_after` | `KGExtractionPhase` enum value that controls how far the pipeline runs (useful for debugging or incremental extraction). | --- ## Completion and status reporting `KgFromDocument` reports status through both the standard skill return value and the shared live progress channel: - `completion_percentage[0]` is initialized to `0.0`, advances through setup and extraction, and is set to `1.0` after the successful result is assembled. Values are fractions from `0.0` to `1.0`. - A successful return contains `"success": True`, together with `community_id`, `entities_count`, and `entities` result data. - A failed return contains `"error": 1` and `"error_message"`. The return dictionary is the authoritative terminal status; progress is only a live indication and is not durable job history. Progress is updated at setup, after community creation, after entity extraction, and after relationship extraction, with additional updates per entity type and relationship type. Early input failures return before the first progress assignment, so callers must use the return dictionary to distinguish failure from an incomplete or stale progress value. The general contract is documented in `SKILLS.md`. ## Document Splitting Two helpers produce different views of the document: | Helper | Output | Used by | |---|---|---| | `_chunk_document()` | Groups consecutive top-level HTML elements into text blocks of ≤ `CHUNK_MAX_CHARS` (3000) characters. Retains `id` attributes so the LLM can emit element-level provenance from within a chunk. | All phases | ### `_chunk_document()` algorithm `strip_html_element_attributes` is applied to each element before size measurement — all HTML attributes **except `id`** are stripped, so element ids survive into the chunk (the LLM uses them to emit `source_id`). ``` 1. Parse the HTML and collect top-level elements. 2. For each top-level element: a. If len(element_html) ≤ CHUNK_MAX_CHARS → add to flat list as-is. b. Else if element has child elements (container) → expand one level: replace the container with its direct child elements (each child added individually to the flat list). c. Else (oversized leaf with no element children) → add as-is regardless of size (no further split is possible without breaking content structure). 3. Accumulate the flat list into character-budget chunks: - Start a new chunk whenever adding the next element would exceed CHUNK_MAX_CHARS. - Join elements within a chunk with '\n'. 4. Fall back to [document_text] as a single chunk if HTML parsing fails or yields no elements. ``` A single LLM call therefore never receives more than ~3000 characters of HTML, except in the leaf-overflow edge case (step 2c). --- ## LLM Input Format Every LLM call is a **multi-message conversation** assembled from the following slots: | # | Role | Content | Present when | |---|---|---|---| | 1 | `system` | Task description, output YAML schema, RULES, and EXAMPLES | Always | | 2 | `system` | Existing entities already in the community (Phase 1) or already-known attribute values (Phase 2) | Only when the community already contains relevant data for the entity type / attribute being extracted | | 3 | `system` | `` — the whole document as **plain text** (HTML stripped via `generate_text_from_html()`). Falls back to the document's stored `summary` if the plain text exceeds `MAX_LOADED_DOCUMENT_LENGTH_DEFAULT`. Absent if both are empty (large document with no summary). | Present for most documents; absent only when `len(plain_text) > MAX_LOADED_DOCUMENT_LENGTH_DEFAULT` and the document has no stored summary | | 4 | `user` | `` — the chunk (Phases 1, 3, 4) or single element (Phase 2) as **HTML with `id` attributes preserved** (all other HTML attributes stripped). This is the only source the LLM is allowed to extract from. | Always | The same `document_context` string (slot 3) is reused unchanged across all LLM calls for a given document run. **Example user message (slot 4):** ```html

Charly, a big fluffy dog with a brown coat, wagged his tail.

Tom handed the letter to Sarah and sat down.

``` The LLM is instructed to copy the `id` attribute of the most specific HTML element containing the evidence into the `source_id` field of its output. --- ## Extraction Phases | Phase | Name | LLM function | Granularity | Output | Notes | |---|---|---|---|---|---| | **1** | Entity Discovery | `_call_llm_extract_entities()` | entity type × chunk | `[{entity_id, name}]` | Name variants collected across chunks and collapsed into `{entity_id, name, names[]}` before persistence. | | **1b** | Entity Deduplication | `_call_llm_deduplicate_entities()` | entity type (no document scan) | `{dedup_map, elements_to_remove}` | Aliases merged under canonical id; wrong-type entries dropped. | | **2** | Entity Attribute Extraction | `_call_llm_extract_entity_attribute()` | entity type × attribute × chunk | `[{entity_id, value, source_id}]` | LLM emits `source_id` pointing to the specific element id within the chunk. All calls for a type dispatched in parallel. | | **3** | Relationship Discovery | `_call_llm_extract_relationships()` | relationship type × chunk | `[{source_entity_id, target_entity_id, source_id}]` | LLM emits `source_id` from inside the chunk. Duplicate `(source, target)` pairs deduplicated in Python; pairs with unknown entity ids discarded. | | **4** | Relationship Attribute Extraction | `_call_llm_extract_relationship_attribute()` | relationship type × attribute × chunk | `[{source_entity_id, target_entity_id, value}]` | Chunk granularity sufficient — relationship identity is fixed by Phase 3; sub-element provenance not needed. | --- ## LLM Call Count Formula ``` Total calls = (E_types × Chunks) -- Phase 1: entity discovery + E_types -- Phase 1b: deduplication + (sum over E_types of Attrs × Chunks) -- Phase 2: entity attributes + (R_types × Chunks) -- Phase 3: relationship discovery + (sum over R_types of Attrs × Chunks) -- Phase 4: relationship attributes ``` Phase 2 can still dominate for ontologies with many attributes and many entity types, but the call count is now bounded by chunks (not elements). ### Example — bob-01.txt with FictionalStories ontology | Variable | Value | |---|---| | Entity types (`E_types`) | 4 | | Total entity attributes | 7 | | Relationship types (`R_types`) | 3 | | Total relationship attributes | 0 | | Chunks | ~4 | $$\underbrace{4 \times 4}_{P1} + \underbrace{4}_{P1b} + \underbrace{7 \times 4}_{P2} + \underbrace{3 \times 4}_{P3} + \underbrace{0}_{P4} = 16 + 4 + 28 + 12 \approx 60 \text{ calls}$$ Previously ~123 calls (Phase 2 was element-based: 7 × 13 = 91). Now ~60 — a ~50% reduction. --- ## Provenance (`source_id`) Every persisted attribute value or relationship carries a `paragraph_id` in Neo4j that traces back to the HTML element where the evidence was found: - **Phase 2**: LLM outputs `source_id` pointing to the specific element id within the chunk. No fallback — chunks have no single element id. - **Phase 3**: LLM outputs `source_id` pointing to the specific element id within the chunk. No fallback. - **Phase 4**: no per-value provenance (relationship identity, not a specific element, is the anchor). --- ## Partial Runs (`KGExtractionPhase`) The `stop_after` parameter accepts a `KGExtractionPhase` enum value: | Value | Runs through | |---|---| | `ENTITIES` | Phase 1 + 1b only | | `ENTITY_ATTRIBUTES` | Phases 1 – 2 | | `RELATIONSHIPS` | Phases 1 – 3 | | `RELATIONSHIP_ATTRIBUTES` | Full run (default) | --- ## Output (Neo4j) All results are written via `KnowledgeGraphEditor`: | Helper | Creates / updates | |---|---| | `entity_create_or_update()` | `:Entity: