Innovator Knowledge Graph Import Pipeline

This document provides a step-by-step guide for extracting, importing, and organizing enterprise models from the Innovator (MIRA) PostgreSQL database into the Thalabus Neo4j Knowledge Graph.

Overview of the Pipeline

The import process consists of four sequential stages:

flowchart LR
    A["Step 1: Ontology Extraction<br/><code>PostgresOntologyExtractor.py</code>"] --> B["Step 2: Nodes & Edges Import<br/><code>ProsgresInnovatorKnowledgeGraphImport.py</code>"]
    B --> C["Step 3: Diagram Image Download<br/><code>DiagramDownloader.py</code>"]
    C --> D["Step 4: Diagram Communities<br/><code>InnovatorDiagramCommunityBuilder.py</code>"]
  1. Step 1 — Ontology Definition: Extracts domain entity stereotypes, baseline/dynamic attributes, relationship endpoint schemas, and sources from PostgreSQL, creating the :Ontology and permanent :Community:Dataset root.
  2. Step 2 — Knowledge Graph Ingestion: Reads elements and edges from PostgreSQL for targeted repositories/models, sanitizes properties, and links all nodes to the parent Community with [:AGGREGATES].
  3. Step 3 — Diagram Image Caching: Fetches SVG diagrams from the Innovator MIRA service, renders them into PNGs, and stores them in the local document repository.
  4. Step 4 — Diagram Community Partitioning: Generates dedicated Community subgraphs for every diagram, connecting the diagram and all elements placed on it via :diagram_occurrences. This same is_diagram/diagram_occurrences pair is later consumed by the source-configuration UI's unified Packages & Diagrams Selection panel and by OntologySynchroniser.py to scope entity/relationship synchronization to selected diagrams (see INTEGRATION_INNOVATOR.md and ONTOLOGY_SYNCHRONISATION.md).

Prerequisites & Environment Configuration

Ensure your .env file contains the required connection parameters for both PostgreSQL (Innovator MIRA) and Neo4j:

# PostgreSQL (Innovator MIRA Database)
INNOVATOR_MIRA_POSTGRESSQL_URL=mid-server.tail443ae8.ts.net
INNOVATOR_MIRA_POSTGRESSQL_USER=innovator
INNOVATOR_MIRA_POSTGRESSQL_PASSWORD=<password>
INNOVATOR_MIRA_POSTGRESSQL_DATABASE=MIRA

# Neo4j Graph Database
NEO4J_URI=bolt://localhost:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=<password>

Step 1: Extract & Import the Ontology

Script: PythonServer/src/core/extensions/skillsets/KnowledgeGraph/PostgresOntologyExtractor.py

Purpose

Inspects PostgreSQL elements and edges tables to extract: - Unique entity stereotypes as :EntityOntology:<Stereotype> nodes. - Standard baseline attributes (name, description, namespace) + dynamic properties from props JSON as :OntologyAttribute nodes. - Distinct relationship types and their source/target stereotype constraints as :RelationshipOntology nodes. - Fully qualified :OntologySource nodes attached via [:HAS_SOURCE]. - Creates the root :Ontology node and its default root :Community:Dataset with the permanent ID DefaultCommunity_Innovator. The ID is the community identifier; no separate Community name is created.

How to Run

Option A: Via VS Code Debugger / Launch Configuration

Select "Python Debugger: Postgres Ontology Extractor" in .vscode/launch.json:

{
    "name": "Python Debugger: Postgres Ontology Extractor",
    "type": "debugpy",
    "request": "launch",
    "module": "src.core.extensions.skillsets.KnowledgeGraph.PostgresOntologyExtractor",
    "cwd": "${workspaceFolder}/PythonServer",
    "args": [
        "--ontology-id", "Innovator",
        "--name", "Innovator",
        "--description", "Innovator Ontology extracted from PostgreSQL",
        "--repository", "TestModelUAKI",
        "--model-uuid", "76414caa-d57e-41e3-924c-deff1a91a17e"
    ]
}

Option B: Via Terminal

cd PythonServer
python -m src.core.extensions.skillsets.KnowledgeGraph.PostgresOntologyExtractor `
    --ontology-id "Innovator" `
    --name "Innovator" `
    --description "Innovator Ontology extracted from PostgreSQL" `
    --repository "TestModelUAKI" `
    --model-uuid "76414caa-d57e-41e3-924c-deff1a91a17e"

Step 2: Import Nodes and Edges from PostgreSQL

Script: PythonServer/src/core/extensions/skillsets/KnowledgeGraph/ProsgresInnovatorKnowledgeGraphImport.py

Purpose

Extracts model elements and edges from PostgreSQL and ingests them into Neo4j as :Entity:Innovator nodes and labeled relationships.

Instructions

  1. Review & Configure Target Models: Open PythonServer/src/core/extensions/skillsets/KnowledgeGraph/ProsgresInnovatorKnowledgeGraphImport.py and configure _MIRA_MODELS to specify which repositories and model UUIDs to import:

python _MIRA_MODELS: List[Dict[str, object]] = [ { "repository": "TestModelUAKI", "modelUuid": "76414caa-d57e-41e3-924c-deff1a91a17e", }, ]

  1. Configure Community ID: Set DEFAULT_COMMUNITY_ID (or pass it as an argument) to match the permanent root community ID created in Step 1: python DEFAULT_COMMUNITY_ID = "DefaultCommunity_Innovator"

How to Run

Option A: Via VS Code Debugger

Select "Python Debugger: Import Postgres Innovator KG" in .vscode/launch.json.

Option B: Via Terminal

cd PythonServer
python -m src.core.extensions.skillsets.KnowledgeGraph.ProsgresInnovatorKnowledgeGraphImport

Step 3: Download & Cache Diagram Images

Script / Skill: PythonServer/src/core/extensions/skillsets/KnowledgeGraph/DiagramDownloader.py

Purpose

Queries Neo4j for all diagram nodes (is_diagram = true), downloads their SVG graphics from the Innovator REST service, renders them to PNG using cairosvg, and caches them in the local document repository (e.g. DocumentRepository/<id>.png).

How to Run

Option A: Via VS Code Debugger

Select "Python Debugger: Diagram Downloader" in .vscode/launch.json:

{
    "name": "Python Debugger: Diagram Downloader",
    "type": "debugpy",
    "request": "launch",
    "module": "src.core.extensions.skillsets.KnowledgeGraph.DiagramDownloader",
    "cwd": "${workspaceFolder}/PythonServer",
    "args": [
        "--repository", "TestModelUAKI",
        "--model-uuid", "76414caa-d57e-41e3-924c-deff1a91a17e"
    ]
}

Option B: Via Terminal

Step 4: Build Diagram Communities

Script: PythonServer/src/core/extensions/skillsets/KnowledgeGraph/InnovatorDiagramCommunityBuilder.py

Purpose

Partitions the knowledge graph by diagram: - Creates a dedicated :Community:Dataset node for every diagram, using an ID such as Innovator_<Diagram_Name>. - Connects the diagram node to the community via [:AGGREGATES]. - Connects all elements appearing on the diagram (via [:diagram_occurrences]) to that community with [:AGGREGATES]. - Fully idempotent: Re-running uses MERGE and will not produce duplicate communities or duplicate relationships.

How to Run

Option A: Via VS Code Debugger

Select "Python Debugger: Innovator Diagram Community Builder" in .vscode/launch.json:

{
    "name": "Python Debugger: Innovator Diagram Community Builder",
    "type": "debugpy",
    "request": "launch",
    "module": "src.core.extensions.skillsets.KnowledgeGraph.InnovatorDiagramCommunityBuilder",
    "cwd": "${workspaceFolder}/PythonServer",
    "args": [
        "--base_community_name", "Innovator",
        "--repository", "TestModelUAKI",
        "--model-uuid", "76414caa-d57e-41e3-924c-deff1a91a17e"
    ]
}

Option B: Via Terminal

cd PythonServer
python -m src.core.extensions.skillsets.KnowledgeGraph.InnovatorDiagramCommunityBuilder `
    --base_community_name "Innovator" `
    --repository "TestModelUAKI" `
    --model-uuid "76414caa-d57e-41e3-924c-deff1a91a17e"

Post-Import Verification Queries in Neo4j

After running all 4 steps, you can execute the following Cypher queries in Neo4j Browser to verify the integrity of the data:

1. Verify Ontology & Communities

MATCH (o:Ontology {id: "Innovator"})-[:CONTAINS]->(e:EntityOntology)
RETURN o.name AS ontology, count(e) AS total_entities;

2. Verify Diagrams and Communities

MATCH (c:Community)
WHERE toLower(c.id) STARTS WITH "innovator_"
OPTIONAL MATCH (c)-[:AGGREGATES]->(n)
RETURN c.id AS community_id, count(n) AS nodes_in_community
ORDER BY nodes_in_community DESC
LIMIT 10;