# Ontology Synchronisation (`OntologySynchroniser`)

## Purpose

`OntologySynchroniser` (`PythonServer/src/core/extensions/skillsets/KnowledgeGraph/OntologySynchroniser.py`) merges data from raw
Neo4j `:Entity` nodes belonging to one ontology (the *source*) into the `:Entity` instances of a different ontology (the *target*),
based on the `kg.<ontology_id>.<entity_label>[:<attribute>]` `OntologySource` mappings created in the ontology editor UI
(`OntologyElement.razor`, see [DATASETS.md](../datasets/DATASETS.md) and the ontology editor documentation for how
`OntologySource` nodes are attached to `EntityOntology`/`OntologyAttribute` nodes).

### Worked example

- Target ontology: `EJPD`, with an `EntityOntology` labeled `GlossaryElement`.
- The `GlossaryElement` entity is mapped to source `kg.Innovator.MeaningGlossaryItem` (an entity-level `OntologySource`), meaning:
  "populate `GlossaryElement` instances from this same Neo4j instance's (`kg`) raw `:Entity` nodes belonging to ontology `Innovator`,
  labeled `MeaningGlossaryItem`".
- Individual `GlossaryElement` attributes (e.g. `Name`) are mapped to `kg.Innovator.MeaningGlossaryItem:name` (an attribute-level
  `OntologySource`), meaning: "populate `GlossaryElement.Name` from `MeaningGlossaryItem.name`".
- Attributes without an explicit `kg.` mapping are **never touched** — left as-is (or empty for new entities).

## Ontology source catalog and type distinction

The `/settingsontologysources` page displays a global catalog of standalone `:OntologySource` nodes. The page loads them with
`MATCH (s:OntologySource)`, so a listed source is not necessarily attached to an ontology element. Each source has these main
properties:

| Property | Meaning |
|---|---|
| `id` | Stable source-definition identifier used by `HAS_SOURCE` and source selection in the UI. |
| `name` | Qualified source name, such as `kg.Innovator.DiagramEAEnterpriseArchitecture` or `Innovator.elements.TestModelUAKI.Diagram`. |
| `type` | Source shape: `Entity`, `Attribute`, or `Relationship`. This is metadata on the source definition, not a Neo4j node label or relationship type. |
| `data_type` | Attribute value type. It is normally `None` for entity and relationship sources. |
| `isBlacklisted` | Whether extraction or source selection should exclude the definition. |

The source definition is attached to the ontology schema with `HAS_SOURCE`:

```cypher
(:Ontology)-[:CONTAINS]->(:EntityOntology)-[:HAS_SOURCE]->(:OntologySource)
(:EntityOntology)-[:HAS_ATTRIBUTE]->(:OntologyAttribute)-[:HAS_SOURCE]->(:OntologySource)
(:Ontology)-[:CONTAINS]->(:RelationshipOntology)-[:HAS_SOURCE]->(:OntologySource)
```

The three source types are modeled differently:

### Entity sources

An `Entity` source identifies a family of source entity instances by name. For internal knowledge-graph sources, the convention is
`kg.<source_ontology_id>.<source_label>`. For Innovator sources, the convention is
`Innovator.elements.<model_id>.<stereotype>`.

The source instances are ordinary `:Entity` nodes selected by their labels. For example, an Innovator Diagram source can select nodes
carrying `:Entity`, `:Innovator`, and the relevant model/stereotype labels. The source catalog node itself is not one of those instances.

### Attribute sources

An `Attribute` source identifies a property mapping and is normally named with an attribute suffix, for example:
`kg.Innovator.DiagramEAEnterpriseArchitecture:name`.

It is attached to an `:OntologyAttribute`, which is attached to an `:EntityOntology` or `:RelationshipOntology` definition:

```cypher
(:EntityOntology)-[:HAS_ATTRIBUTE]->(:OntologyAttribute)-[:HAS_SOURCE]->(:OntologySource {type: 'Attribute'})
```

The attribute source does not represent a separate Neo4j node instance. Consequently, the source table displays `-` for its instance
count. During synchronization, the mapped source property is copied into the target entity's attributes and tagged with
`kg://<source_entity_id>` provenance.

### Relationship sources

A `Relationship` source identifies a relationship type rather than an entity-label pair. Internal names use
`kg.<source_ontology_id>.<relationship_type>` and Innovator names use
`Innovator.edges.<model_id>.<relationship_type>`.

The source instances are Neo4j relationships whose runtime type matches the source name's relationship type:

```cypher
MATCH ()-[r]->()
WHERE type(r) = $relationshipType
RETURN count(r)
```

The relationship ontology definition is a `:RelationshipOntology` node, and its source definition is attached with `HAS_SOURCE`.
After entity synchronisation completes, `OntologySynchroniser` also synchronises target relationships for every
`:RelationshipOntology` definition, using its `source_relationship_types` (the raw Innovator relationship type names selected in
the Relationships Selection panel) as the source instances to copy — see [Relationship synchronisation](#relationship-synchronisation).

### Linked source definitions

When an ontology source has linked entity sources, the catalog represents those links separately:

```cypher
(:OntologySource)-[:REFER_TO]->(:OntologySource)
```

`REFER_TO` is used for source-definition links, such as an attribute with `data_type = 'Link'`. It is not the relationship connecting an
ontology element to its source; that connection is always `HAS_SOURCE`.

### Example: EJPD Diagram mapping

The EJPD Diagram mapping therefore has this schema shape:

```cypher
(:Ontology {id: 'EJPD'})
  -[:CONTAINS]->(:EntityOntology:Diagram)
  -[:HAS_SOURCE]->(:OntologySource {name: 'kg.Innovator.DiagramDMClassDomain', type: 'Entity'})

(:Ontology {id: 'EJPD'})
  -[:CONTAINS]->(:EntityOntology:Diagram)
  -[:HAS_SOURCE]->(:OntologySource {name: 'kg.Innovator.DiagramEAEnterpriseArchitecture', type: 'Entity'})
```

The two source definitions are used to count the matching raw `:Entity` nodes (one source contributes 1 and the other 47 in the
observed graph). They are not directly connected to those 48 raw nodes by `HAS_SOURCE` or another source-instance relationship.

## Design principles

1. **Idempotency** — running the skill multiple times with unchanged source data produces the same end state (no duplicate
   values, no duplicate nodes).
2. **Non-destructive** — the target ontology's existing `:Entity` nodes are never deleted before or during synchronisation.
3. **MERGE, never CREATE** — every write is a Cypher `MERGE`, so re-running is always safe.
4. **Explicit mapping only** — an attribute is copied only if it has an explicit `kg.<ontology_id>.<label>:<attribute>`
   `OntologySource` attached in the target ontology's schema.
5. **Immediate primary-key consolidation** — attributes forming a target label's primary key (per
   `OntologyService.get_primary_key_attribute_names(label)`) have both their raw source-list value **and** their
   `<attribute>_consolidated` value written immediately, bypassing the normal asynchronous consolidation loop
   (`Maintenance.py`'s periodic `consolidate_knowledge_graph()`). Non-key attributes are merged as regular source-tagged values
   and flagged with `requires_consolidation = true` for the existing periodic consolidation pass.
6. **Shared consolidation status** — the skill reports `IsConsolidationInProgress = true` through the Blazor controller when it
  starts and whenever it marks an Entity with `requires_consolidation = true`. The maintenance thread recomputes the remaining
  flagged Entity count after each pass and reports false only when none remain. The shared header displays this state to power users
  and platform administrators.

## Algorithm

For each target label (either the `selected_labels` argument, comma-separated, or — if empty — every non-reserved
`EntityOntology` label under `ontology_id`):

1. **Resolve the entity-level source.** Find the `kg.<source_ontology_id>.<source_label>` `OntologySource` attached to the
   target `EntityOntology` node via `HAS_SOURCE`. If none is found, the label is skipped (nothing to synchronise from).
2. **Resolve the attribute mapping.** For every `OntologyAttribute` of the target label, find its attached
   `kg.<source_ontology_id>.<source_label>:<source_attribute>` `OntologySource` and build a
   `target_attribute -> source_attribute` dict. Attributes without such a source are excluded from the mapping (and therefore
   never written).
3. **Fetch primary-key attribute names** for the target label via `get_ontology_service().get_primary_key_attribute_names(label)`.
4. **Fetch source instances.** Raw source `:Entity` nodes are matched by Neo4j label and filtered through the target ontology's
  explicitly selected Innovator packages **and diagrams** (selected in the unified Packages & Diagrams Selection panel):
  `(:Ontology)-[:SYNCHRONIZE_SOURCE]->(selected:Entity:Innovator)`, where `selected` is a package or a diagram
  (`is_diagram = true`). A source entity `e` qualifies when `(e)-[:owner*1..]->(selected)` (descendant of a selected package),
  or `(e)-[:diagram_occurrences]->(selected)` (an element placed on a selected diagram), or `e = selected` (the diagram entity
  itself). This includes direct elements and descendants of each selected package, plus the diagram entity and every element
  occurring on each selected diagram, while excluding elements outside these scopes. Partially selected package states are
  derived in the UI and are never persisted, so they cannot broaden synchronization. The source ontology id doubles as a
  Neo4j label on those nodes (e.g. `:Entity:Innovator:MeaningGlossaryItem`), consistent with how `InnovatorKnowledgeGraphImport.py`
  creates them.
5. **For each source instance:**
   - Collect the primary-key attribute values by reading, for every mapped primary-key attribute, the source's
     `<source_attribute>_consolidated` property (falling back to the first value found in the source's raw JSON source-list
     property if the consolidated value is missing). If any mapped primary-key value cannot be resolved, the instance is
     skipped (logged as a warning) — a partial key cannot be used to safely identify/deduplicate the target entity.
   - Compute a **deterministic target entity id**: `tokenize_entity_id(f"{label}_{'|'.join(primary_key_values)}")`
     (or `tokenize_entity_id(f"{label}_{source_entity_id}")` if the label has no primary key). Because the id is derived
     directly from the primary-key value(s), re-synchronising the same source data always resolves to the same target node —
     this is what gives `MERGE (e:Entity:<label> {id: $entity_id})` the same deduplication effect as matching directly on
     `<primary_key>_consolidated`, while keeping the existing id-based `:Entity` model used everywhere else in the codebase.
   - Build the new attribute values: each mapped attribute's raw source value is copied and **re-tagged** with a fresh
     `source_id` of the form `kg://<source_entity_id>` (the source's own internal source_id chain — e.g. document/paragraph
     references — is intentionally not carried over; the target's source_id instead traces back to the *contributing kg
     entity itself*).
   - Merge these new attribute values into the target entity's existing attributes using the same
     `merge_attribute()` / `merge_attributes()` helpers used by `KnowledgeGraphEditor.entity_create_or_update()` (see
     `KnowledgeGraphEditor.py`), which already deduplicate identical value+source combinations — this is what makes repeated
     runs idempotent.
   - `MERGE (e:Entity:<label> {id: $entity_id}) SET ...` writing:
     - the merged raw attribute values (JSON-encoded source lists), and
     - for primary-key attributes only, the `<attribute>_consolidated` value, written immediately, and
     - `requires_consolidation = true` whenever any attribute actually changed (so the periodic Maintenance thread
       recomputes `_consolidated` values and `aggregate_values` for non-key attributes as usual).
   - `MERGE (c:Community:Dataset {id: $community_id})-[:AGGREGATES]->(e)` to link the merged entity into the target ontology's
     default community. The permanent community ID is `DefaultCommunity_<ontology_id>`, created/ensured — never recreated
     destructively — using the same ID convention as `PostgresOntologyExtractor._ensure_ontology_and_community()`. The
     community has no separate `name` property, so this ID is also the value used for lookup, navigation, and rename operations.
     This lets it participate in the same dataset/community-scoped views (search, listing, ACL) as normally-extracted entities.
   - `MERGE (target)-[:SOURCE]->(source)` to preserve the direct provenance link to the raw source `:Entity`, including when the
     target already exists or has no mapped attributes. Multiple source links are allowed.

## Relationship synchronisation

Once every selected entity label has been synchronised, `OntologySynchroniser` synchronises relationships for **all**
`:RelationshipOntology` definitions under the target ontology, regardless of the `selected_labels` argument (that argument scopes
only the entity phase). This is intentional: relationship endpoints may already have been synchronised in a previous run, and
package selection alone is sufficient to scope which relationship instances qualify.

For each `:RelationshipOntology` definition with at least one `source_relationship_types` entry (the raw Innovator relationship
type names, set from the Relationships Selection panel — see
[Ontology Synchronization Counts](../../../Documentation/Documents/ONTOLOGY_SYNCHRONIZATION_COUNTS.md)), and for each such raw
relationship type:

1. **Fetch source relationship instances**, scoped to selected packages/diagrams on **both** endpoints:
   `MATCH (source:Entity:Innovator)-[:\`<type>\`]->(target:Entity:Innovator)` where both `source` and `target` are non-structure and
   each independently satisfies `(entity)-[:owner*1..]->(selected)` or `(entity)-[:diagram_occurrences]->(selected)` or
   `entity = selected`, for some `(:Ontology)-[:SYNCHRONIZE_SOURCE]->(selected)` (a selected package or diagram).
   A relationship whose source or target falls outside every selected package subtree and every selected diagram's occurrences
   is skipped.
2. **Resolve each endpoint's already-synchronised target Entity id.** For each of the raw endpoint's non-reserved Neo4j labels, the
   label is looked up against a reverse index of every target label's entity-level `kg.` source (built once per run, across *all*
   target labels, not just `selected_labels`) to find a candidate target label, then the same deterministic id formula used by
   entity sync (`tokenize_entity_id(f"{target_label}_{...}")`, from primary-key values or the raw entity id) is recomputed. If no
   label resolves, the endpoint is considered never-synchronised and the relationship instance is skipped — this phase never
   creates target `:Entity` nodes, it only connects entities that already exist.
3. **Merge the target relationship**, only once both endpoints resolve: `MATCH (e1:Entity {id: $source_entity_id}) MATCH
   (e2:Entity {id: $target_entity_id}) MERGE (e1)-[rel:\`<target_relationship_label>\`]->(e2) SET rel.source_id = 'kg://<source_id>-><target_id>'`.
   The target relationship type is the `:RelationshipOntology` definition's own label (the same label used when reading community
   relationships, e.g. in `Ontology._get_relationships_as_string`).

## Non-goals / explicit exclusions

- Attributes without an explicit `kg.` `OntologySource` mapping are **never** read, written, or cleared.
- The skill never deletes target `:Entity` nodes, even if their corresponding source instance is later removed.
- Relationship synchronisation never creates target `:Entity` nodes; both endpoints must already have been synchronised by the
  entity phase (in this run or a previous one).

## Key functions (`OntologySynchroniser.py`)

| Function | Responsibility |
|---|---|
| `OntologySynchroniser(ontology_id, selected_labels="")` | Entry point / skill function. Runs the entity phase (scoped by `selected_labels`) then the relationship phase (unscoped), reports `completion_percentage`, returns `{success, error_message}`. |
| `_get_target_labels` | Resolves the list of target labels: explicit selection, or all non-reserved `EntityOntology` labels under `ontology_id`. |
| `_get_entity_kg_source` | Resolves the entity-level `kg.<source_ontology_id>.<source_label>` mapping for a target label. |
| `_get_attribute_kg_mapping` | Resolves the `target_attribute -> source_attribute` mapping for a target label's attributes. |
| `_ensure_target_community` | Ensures (MERGE, non-destructive) the permanent `:Community:Dataset` with ID `DefaultCommunity_<ontology_id>` exists; it does not create a separate `name` property. |
| `_sync_target_label` | Orchestrates synchronisation of all source instances for one target label. |
| `_sync_source_entity` | Computes the deterministic target id and primary-key values for one source instance, then merges it. |
| `_merge_target_entity` | Executes the `MERGE ... SET` writing merged attributes and immediately-consolidated primary-key values. |
| `_link_entity_to_source` | `MERGE`s the synchronized target `:Entity` to its raw source `:Entity` with `[:SOURCE]`, preserving provenance independently of attributes. |
| `_link_entity_to_community` | `MERGE`s the `(:Community)-[:AGGREGATES]->(:Entity)` relationship. |
| `_parse_kg_source_name` | Parses a `kg.<ontology_id>.<label>[:<attribute>]` `OntologySource` name via regex. |
| `_extract_first_value` / `_build_new_attribute_value` | Helpers for reading/re-tagging raw JSON source-list attribute values. |
| `_get_relationship_definitions` | Resolves every `:RelationshipOntology` definition with a mapped `source_relationship_types` list. |
| `_build_entity_source_reverse_map` | Builds a `(source_ontology_id, source_label) -> target_label` index across all target labels. |
| `_build_target_label_mapping_cache` | Precomputes primary-key names and attribute mapping per target label, reused when resolving relationship endpoints. |
| `_resolve_target_entity_id` | Resolves the deterministic target Entity id already merged for one raw source endpoint, or `None` if never synchronised. |
| `_sync_relationship_type` | Merges target relationships for one raw relationship type, scoped to selected packages on both endpoints. |
| `_sync_relationships` | Orchestrates relationship synchronisation across every `:RelationshipOntology` definition. |

## Invocation

Triggered from the UI via `ThalabusCore/Data/Jobs/OntologySynchronizationJob.cs`, which passes `ontology_id` and
`selected_labels` as skill arguments and reports `completion_percentage` back to the job's progress bar
(see [SettingsOntologySynchronization.razor](../../../ThalabusWeb/Pages/SettingsPages/SettingsOntologySynchronization.razor)).

Can also be run standalone for testing:

```powershell
python -m src.core.extensions.skillsets.KnowledgeGraph.OntologySynchroniser
```

(the `if __name__ == "__main__":` block invokes it with `ontology_id="test-ontology"`, `selected_labels="GlossaryElement"`).

## Related documentation

- [KG Extraction Pipeline](KG_EXTRACTION.md) — how raw `:Entity` nodes are originally extracted/populated from documents, and the
  source-list attribute storage convention (`{"value": ..., "source_id": ...}`) reused by this skill.
- [Extraction Links Maintenance](EXTRACTION_LINKS_MAINTENANCE.md) — background maintenance thread and URI reference conventions
  (`doc://`, `kg://`, `inno://`) that this skill's `kg://<element_id>` source references are consistent with.