Ontology Synchronisation (OntologySynchroniser)

Purpose

OntologySynchroniser (PythonServer/src/core/extensions/skillsets/KnowledgeGraph/OntologySynchroniser.py) merges data from raw Neo4j :Entity nodes belonging to one ontology (the source) into the :Entity instances of a different ontology (the target), based on the kg.<ontology_id>.<entity_label>[:<attribute>] OntologySource mappings created in the ontology editor UI (OntologyElement.razor, see DATASETS.md and the ontology editor documentation for how OntologySource nodes are attached to EntityOntology/OntologyAttribute nodes).

Worked example

Ontology source catalog and type distinction

The /settingsontologysources page displays a global catalog of standalone :OntologySource nodes. The page loads them with MATCH (s:OntologySource), so a listed source is not necessarily attached to an ontology element. Each source has these main properties:

Property Meaning
id Stable source-definition identifier used by HAS_SOURCE and source selection in the UI.
name Qualified source name, such as kg.Innovator.DiagramEAEnterpriseArchitecture or Innovator.elements.TestModelUAKI.Diagram.
type Source shape: Entity, Attribute, or Relationship. This is metadata on the source definition, not a Neo4j node label or relationship type.
data_type Attribute value type. It is normally None for entity and relationship sources.
isBlacklisted Whether extraction or source selection should exclude the definition.

The source definition is attached to the ontology schema with HAS_SOURCE:

(:Ontology)-[:CONTAINS]->(:EntityOntology)-[:HAS_SOURCE]->(:OntologySource)
(:EntityOntology)-[:HAS_ATTRIBUTE]->(:OntologyAttribute)-[:HAS_SOURCE]->(:OntologySource)
(:Ontology)-[:CONTAINS]->(:RelationshipOntology)-[:HAS_SOURCE]->(:OntologySource)

The three source types are modeled differently:

Entity sources

An Entity source identifies a family of source entity instances by name. For internal knowledge-graph sources, the convention is kg.<source_ontology_id>.<source_label>. For Innovator sources, the convention is Innovator.elements.<model_id>.<stereotype>.

The source instances are ordinary :Entity nodes selected by their labels. For example, an Innovator Diagram source can select nodes carrying :Entity, :Innovator, and the relevant model/stereotype labels. The source catalog node itself is not one of those instances.

Attribute sources

An Attribute source identifies a property mapping and is normally named with an attribute suffix, for example: kg.Innovator.DiagramEAEnterpriseArchitecture:name.

It is attached to an :OntologyAttribute, which is attached to an :EntityOntology or :RelationshipOntology definition:

(:EntityOntology)-[:HAS_ATTRIBUTE]->(:OntologyAttribute)-[:HAS_SOURCE]->(:OntologySource {type: 'Attribute'})

The attribute source does not represent a separate Neo4j node instance. Consequently, the source table displays - for its instance count. During synchronization, the mapped source property is copied into the target entity's attributes and tagged with kg://<source_entity_id> provenance.

Relationship sources

A Relationship source identifies a relationship type rather than an entity-label pair. Internal names use kg.<source_ontology_id>.<relationship_type> and Innovator names use Innovator.edges.<model_id>.<relationship_type>.

The source instances are Neo4j relationships whose runtime type matches the source name's relationship type:

MATCH ()-[r]->()
WHERE type(r) = $relationshipType
RETURN count(r)

The relationship ontology definition is a :RelationshipOntology node, and its source definition is attached with HAS_SOURCE. After entity synchronisation completes, OntologySynchroniser also synchronises target relationships for every :RelationshipOntology definition, using its source_relationship_types (the raw Innovator relationship type names selected in the Relationships Selection panel) as the source instances to copy — see Relationship synchronisation.

Linked source definitions

When an ontology source has linked entity sources, the catalog represents those links separately:

(:OntologySource)-[:REFER_TO]->(:OntologySource)

REFER_TO is used for source-definition links, such as an attribute with data_type = 'Link'. It is not the relationship connecting an ontology element to its source; that connection is always HAS_SOURCE.

Example: EJPD Diagram mapping

The EJPD Diagram mapping therefore has this schema shape:

(:Ontology {id: 'EJPD'})
  -[:CONTAINS]->(:EntityOntology:Diagram)
  -[:HAS_SOURCE]->(:OntologySource {name: 'kg.Innovator.DiagramDMClassDomain', type: 'Entity'})

(:Ontology {id: 'EJPD'})
  -[:CONTAINS]->(:EntityOntology:Diagram)
  -[:HAS_SOURCE]->(:OntologySource {name: 'kg.Innovator.DiagramEAEnterpriseArchitecture', type: 'Entity'})

The two source definitions are used to count the matching raw :Entity nodes (one source contributes 1 and the other 47 in the observed graph). They are not directly connected to those 48 raw nodes by HAS_SOURCE or another source-instance relationship.

Design principles

  1. Idempotency — running the skill multiple times with unchanged source data produces the same end state (no duplicate values, no duplicate nodes).
  2. Non-destructive — the target ontology's existing :Entity nodes are never deleted before or during synchronisation.
  3. MERGE, never CREATE — every write is a Cypher MERGE, so re-running is always safe.
  4. Explicit mapping only — an attribute is copied only if it has an explicit kg.<ontology_id>.<label>:<attribute> OntologySource attached in the target ontology's schema.
  5. Immediate primary-key consolidation — attributes forming a target label's primary key (per OntologyService.get_primary_key_attribute_names(label)) have both their raw source-list value and their <attribute>_consolidated value written immediately, bypassing the normal asynchronous consolidation loop (Maintenance.py's periodic consolidate_knowledge_graph()). Non-key attributes are merged as regular source-tagged values and flagged with requires_consolidation = true for the existing periodic consolidation pass.
  6. Shared consolidation status — the skill reports IsConsolidationInProgress = true through the Blazor controller when it starts and whenever it marks an Entity with requires_consolidation = true. The maintenance thread recomputes the remaining flagged Entity count after each pass and reports false only when none remain. The shared header displays this state to power users and platform administrators.

Algorithm

For each target label (either the selected_labels argument, comma-separated, or — if empty — every non-reserved EntityOntology label under ontology_id):

  1. Resolve the entity-level source. Find the kg.<source_ontology_id>.<source_label> OntologySource attached to the target EntityOntology node via HAS_SOURCE. If none is found, the label is skipped (nothing to synchronise from).
  2. Resolve the attribute mapping. For every OntologyAttribute of the target label, find its attached kg.<source_ontology_id>.<source_label>:<source_attribute> OntologySource and build a target_attribute -> source_attribute dict. Attributes without such a source are excluded from the mapping (and therefore never written).
  3. Fetch primary-key attribute names for the target label via get_ontology_service().get_primary_key_attribute_names(label).
  4. Fetch source instances. Raw source :Entity nodes are matched by Neo4j label and filtered through the target ontology's explicitly selected Innovator packages and diagrams (selected in the unified Packages & Diagrams Selection panel): (:Ontology)-[:SYNCHRONIZE_SOURCE]->(selected:Entity:Innovator), where selected is a package or a diagram (is_diagram = true). A source entity e qualifies when (e)-[:owner*1..]->(selected) (descendant of a selected package), or (e)-[:diagram_occurrences]->(selected) (an element placed on a selected diagram), or e = selected (the diagram entity itself). This includes direct elements and descendants of each selected package, plus the diagram entity and every element occurring on each selected diagram, while excluding elements outside these scopes. Partially selected package states are derived in the UI and are never persisted, so they cannot broaden synchronization. The source ontology id doubles as a Neo4j label on those nodes (e.g. :Entity:Innovator:MeaningGlossaryItem), consistent with how InnovatorKnowledgeGraphImport.py creates them.
  5. For each source instance:
  6. Collect the primary-key attribute values by reading, for every mapped primary-key attribute, the source's <source_attribute>_consolidated property (falling back to the first value found in the source's raw JSON source-list property if the consolidated value is missing). If any mapped primary-key value cannot be resolved, the instance is skipped (logged as a warning) — a partial key cannot be used to safely identify/deduplicate the target entity.
  7. Compute a deterministic target entity id: tokenize_entity_id(f"{label}_{'|'.join(primary_key_values)}") (or tokenize_entity_id(f"{label}_{source_entity_id}") if the label has no primary key). Because the id is derived directly from the primary-key value(s), re-synchronising the same source data always resolves to the same target node — this is what gives MERGE (e:Entity:<label> {id: $entity_id}) the same deduplication effect as matching directly on <primary_key>_consolidated, while keeping the existing id-based :Entity model used everywhere else in the codebase.
  8. Build the new attribute values: each mapped attribute's raw source value is copied and re-tagged with a fresh source_id of the form kg://<source_entity_id> (the source's own internal source_id chain — e.g. document/paragraph references — is intentionally not carried over; the target's source_id instead traces back to the contributing kg entity itself).
  9. Merge these new attribute values into the target entity's existing attributes using the same merge_attribute() / merge_attributes() helpers used by KnowledgeGraphEditor.entity_create_or_update() (see KnowledgeGraphEditor.py), which already deduplicate identical value+source combinations — this is what makes repeated runs idempotent.
  10. MERGE (e:Entity:<label> {id: $entity_id}) SET ... writing:
    • the merged raw attribute values (JSON-encoded source lists), and
    • for primary-key attributes only, the <attribute>_consolidated value, written immediately, and
    • requires_consolidation = true whenever any attribute actually changed (so the periodic Maintenance thread recomputes _consolidated values and aggregate_values for non-key attributes as usual).
  11. MERGE (c:Community:Dataset {id: $community_id})-[:AGGREGATES]->(e) to link the merged entity into the target ontology's default community. The permanent community ID is DefaultCommunity_<ontology_id>, created/ensured — never recreated destructively — using the same ID convention as PostgresOntologyExtractor._ensure_ontology_and_community(). The community has no separate name property, so this ID is also the value used for lookup, navigation, and rename operations. This lets it participate in the same dataset/community-scoped views (search, listing, ACL) as normally-extracted entities.
  12. MERGE (target)-[:SOURCE]->(source) to preserve the direct provenance link to the raw source :Entity, including when the target already exists or has no mapped attributes. Multiple source links are allowed.

Relationship synchronisation

Once every selected entity label has been synchronised, OntologySynchroniser synchronises relationships for all :RelationshipOntology definitions under the target ontology, regardless of the selected_labels argument (that argument scopes only the entity phase). This is intentional: relationship endpoints may already have been synchronised in a previous run, and package selection alone is sufficient to scope which relationship instances qualify.

For each :RelationshipOntology definition with at least one source_relationship_types entry (the raw Innovator relationship type names, set from the Relationships Selection panel — see Ontology Synchronization Counts), and for each such raw relationship type:

  1. Fetch source relationship instances, scoped to selected packages/diagrams on both endpoints: MATCH (source:Entity:Innovator)-[:\`]->(target:Entity:Innovator)where bothsourceandtargetare non-structure and each independently satisfies(entity)-[:owner*1..]->(selected)or(entity)-[:diagram_occurrences]->(selected)orentity = selected, for some(:Ontology)-[:SYNCHRONIZE_SOURCE]->(selected)` (a selected package or diagram). A relationship whose source or target falls outside every selected package subtree and every selected diagram's occurrences is skipped.
  2. Resolve each endpoint's already-synchronised target Entity id. For each of the raw endpoint's non-reserved Neo4j labels, the label is looked up against a reverse index of every target label's entity-level kg. source (built once per run, across all target labels, not just selected_labels) to find a candidate target label, then the same deterministic id formula used by entity sync (tokenize_entity_id(f"{target_label}_{...}"), from primary-key values or the raw entity id) is recomputed. If no label resolves, the endpoint is considered never-synchronised and the relationship instance is skipped — this phase never creates target :Entity nodes, it only connects entities that already exist.
  3. Merge the target relationship, only once both endpoints resolve: MATCH (e1:Entity {id: $source_entity_id}) MATCH (e2:Entity {id: $target_entity_id}) MERGE (e1)-[rel:\`]->(e2) SET rel.source_id = 'kg://->'. The target relationship type is the:RelationshipOntologydefinition's own label (the same label used when reading community relationships, e.g. inOntology._get_relationships_as_string`).

Non-goals / explicit exclusions

Key functions (OntologySynchroniser.py)

Function Responsibility
OntologySynchroniser(ontology_id, selected_labels="") Entry point / skill function. Runs the entity phase (scoped by selected_labels) then the relationship phase (unscoped), reports completion_percentage, returns {success, error_message}.
_get_target_labels Resolves the list of target labels: explicit selection, or all non-reserved EntityOntology labels under ontology_id.
_get_entity_kg_source Resolves the entity-level kg.<source_ontology_id>.<source_label> mapping for a target label.
_get_attribute_kg_mapping Resolves the target_attribute -> source_attribute mapping for a target label's attributes.
_ensure_target_community Ensures (MERGE, non-destructive) the permanent :Community:Dataset with ID DefaultCommunity_<ontology_id> exists; it does not create a separate name property.
_sync_target_label Orchestrates synchronisation of all source instances for one target label.
_sync_source_entity Computes the deterministic target id and primary-key values for one source instance, then merges it.
_merge_target_entity Executes the MERGE ... SET writing merged attributes and immediately-consolidated primary-key values.
_link_entity_to_source MERGEs the synchronized target :Entity to its raw source :Entity with [:SOURCE], preserving provenance independently of attributes.
_link_entity_to_community MERGEs the (:Community)-[:AGGREGATES]->(:Entity) relationship.
_parse_kg_source_name Parses a kg.<ontology_id>.<label>[:<attribute>] OntologySource name via regex.
_extract_first_value / _build_new_attribute_value Helpers for reading/re-tagging raw JSON source-list attribute values.
_get_relationship_definitions Resolves every :RelationshipOntology definition with a mapped source_relationship_types list.
_build_entity_source_reverse_map Builds a (source_ontology_id, source_label) -> target_label index across all target labels.
_build_target_label_mapping_cache Precomputes primary-key names and attribute mapping per target label, reused when resolving relationship endpoints.
_resolve_target_entity_id Resolves the deterministic target Entity id already merged for one raw source endpoint, or None if never synchronised.
_sync_relationship_type Merges target relationships for one raw relationship type, scoped to selected packages on both endpoints.
_sync_relationships Orchestrates relationship synchronisation across every :RelationshipOntology definition.

Invocation

Triggered from the UI via ThalabusCore/Data/Jobs/OntologySynchronizationJob.cs, which passes ontology_id and selected_labels as skill arguments and reports completion_percentage back to the job's progress bar (see SettingsOntologySynchronization.razor).

Can also be run standalone for testing:

python -m src.core.extensions.skillsets.KnowledgeGraph.OntologySynchroniser

(the if __name__ == "__main__": block invokes it with ontology_id="test-ontology", selected_labels="GlossaryElement").

Related documentation