# Innovator Knowledge Graph Import Pipeline

This document provides a step-by-step guide for extracting, importing, and organizing enterprise models from the Innovator (MIRA) PostgreSQL database into the Thalabus Neo4j Knowledge Graph.

---

## Overview of the Pipeline

The import process consists of four sequential stages:

```mermaid
flowchart LR
    A["Step 1: Ontology Extraction<br/><code>PostgresOntologyExtractor.py</code>"] --> B["Step 2: Nodes & Edges Import<br/><code>ProsgresInnovatorKnowledgeGraphImport.py</code>"]
    B --> C["Step 3: Diagram Image Download<br/><code>DiagramDownloader.py</code>"]
    C --> D["Step 4: Diagram Communities<br/><code>InnovatorDiagramCommunityBuilder.py</code>"]
```

1. **Step 1 — Ontology Definition**: Extracts domain entity stereotypes, baseline/dynamic attributes, relationship endpoint schemas, and sources from PostgreSQL, creating the `:Ontology` and permanent `:Community:Dataset` root.
2. **Step 2 — Knowledge Graph Ingestion**: Reads elements and edges from PostgreSQL for targeted repositories/models, sanitizes properties, and links all nodes to the parent Community with `[:AGGREGATES]`.
3. **Step 3 — Diagram Image Caching**: Fetches SVG diagrams from the Innovator MIRA service, renders them into PNGs, and stores them in the local document repository.
4. **Step 4 — Diagram Community Partitioning**: Generates dedicated Community subgraphs for every diagram, connecting the diagram and all elements placed on it via `:diagram_occurrences`. This same `is_diagram`/`diagram_occurrences` pair is later consumed by the source-configuration UI's unified Packages & Diagrams Selection panel and by `OntologySynchroniser.py` to scope entity/relationship synchronization to selected diagrams (see [INTEGRATION_INNOVATOR.md](INTEGRATION_INNOVATOR.md) and [ONTOLOGY_SYNCHRONISATION.md](../ontology/ONTOLOGY_SYNCHRONISATION.md)).

---

## Prerequisites & Environment Configuration

Ensure your `.env` file contains the required connection parameters for both PostgreSQL (Innovator MIRA) and Neo4j:

```dotenv
# PostgreSQL (Innovator MIRA Database)
INNOVATOR_MIRA_POSTGRESSQL_URL=mid-server.tail443ae8.ts.net
INNOVATOR_MIRA_POSTGRESSQL_USER=innovator
INNOVATOR_MIRA_POSTGRESSQL_PASSWORD=<password>
INNOVATOR_MIRA_POSTGRESSQL_DATABASE=MIRA

# Neo4j Graph Database
NEO4J_URI=bolt://localhost:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=<password>
```

---

## Step 1: Extract & Import the Ontology

**Script**: `PythonServer/src/core/extensions/skillsets/KnowledgeGraph/PostgresOntologyExtractor.py`

### Purpose
Inspects PostgreSQL `elements` and `edges` tables to extract:
- Unique entity **stereotypes** as `:EntityOntology:<Stereotype>` nodes.
- Standard baseline attributes (`name`, `description`, `namespace`) + dynamic properties from `props` JSON as `:OntologyAttribute` nodes.
- Distinct relationship types and their source/target stereotype constraints as `:RelationshipOntology` nodes.
- Fully qualified `:OntologySource` nodes attached via `[:HAS_SOURCE]`.
- Creates the root `:Ontology` node and its default root `:Community:Dataset` with the permanent ID
    `DefaultCommunity_Innovator`. The ID is the community identifier; no separate Community `name` is created.

### How to Run

#### Option A: Via VS Code Debugger / Launch Configuration
Select **"Python Debugger: Postgres Ontology Extractor"** in `.vscode/launch.json`:
```json
{
    "name": "Python Debugger: Postgres Ontology Extractor",
    "type": "debugpy",
    "request": "launch",
    "module": "src.core.extensions.skillsets.KnowledgeGraph.PostgresOntologyExtractor",
    "cwd": "${workspaceFolder}/PythonServer",
    "args": [
        "--ontology-id", "Innovator",
        "--name", "Innovator",
        "--description", "Innovator Ontology extracted from PostgreSQL",
        "--repository", "TestModelUAKI",
        "--model-uuid", "76414caa-d57e-41e3-924c-deff1a91a17e"
    ]
}
```

#### Option B: Via Terminal
```powershell
cd PythonServer
python -m src.core.extensions.skillsets.KnowledgeGraph.PostgresOntologyExtractor `
    --ontology-id "Innovator" `
    --name "Innovator" `
    --description "Innovator Ontology extracted from PostgreSQL" `
    --repository "TestModelUAKI" `
    --model-uuid "76414caa-d57e-41e3-924c-deff1a91a17e"
```

---

## Step 2: Import Nodes and Edges from PostgreSQL

**Script**: `PythonServer/src/core/extensions/skillsets/KnowledgeGraph/ProsgresInnovatorKnowledgeGraphImport.py`

### Purpose
Extracts model elements and edges from PostgreSQL and ingests them into Neo4j as `:Entity:Innovator` nodes and labeled relationships.

### Instructions

1. **Review & Configure Target Models**:
   Open `PythonServer/src/core/extensions/skillsets/KnowledgeGraph/ProsgresInnovatorKnowledgeGraphImport.py` and configure `_MIRA_MODELS` to specify which repositories and model UUIDs to import:

   ```python
   _MIRA_MODELS: List[Dict[str, object]] = [
       {
           "repository": "TestModelUAKI",
           "modelUuid": "76414caa-d57e-41e3-924c-deff1a91a17e",
       },
   ]
   ```

2. **Configure Community ID**:
    Set `DEFAULT_COMMUNITY_ID` (or pass it as an argument) to match the permanent root community ID created in Step 1:
   ```python
   DEFAULT_COMMUNITY_ID = "DefaultCommunity_Innovator"
   ```

### How to Run

#### Option A: Via VS Code Debugger
Select **"Python Debugger: Import Postgres Innovator KG"** in `.vscode/launch.json`.

#### Option B: Via Terminal
```powershell
cd PythonServer
python -m src.core.extensions.skillsets.KnowledgeGraph.ProsgresInnovatorKnowledgeGraphImport
```

---

## Step 3: Download & Cache Diagram Images

**Script / Skill**: `PythonServer/src/core/extensions/skillsets/KnowledgeGraph/DiagramDownloader.py`

### Purpose
Queries Neo4j for all diagram nodes (`is_diagram = true`), downloads their SVG graphics from the Innovator REST service, renders them to PNG using `cairosvg`, and caches them in the local document repository (e.g. `DocumentRepository/<id>.png`).

### How to Run

#### Option A: Via VS Code Debugger
Select **"Python Debugger: Diagram Downloader"** in `.vscode/launch.json`:
```json
{
    "name": "Python Debugger: Diagram Downloader",
    "type": "debugpy",
    "request": "launch",
    "module": "src.core.extensions.skillsets.KnowledgeGraph.DiagramDownloader",
    "cwd": "${workspaceFolder}/PythonServer",
    "args": [
        "--repository", "TestModelUAKI",
        "--model-uuid", "76414caa-d57e-41e3-924c-deff1a91a17e"
    ]
}
```

#### Option B: Via Terminal
- **Download for a specific repository & model**:
  ```powershell
  cd PythonServer
  python -m src.core.extensions.skillsets.KnowledgeGraph.DiagramDownloader `
      --repository "TestModelUAKI" `
      --model-uuid "76414caa-d57e-41e3-924c-deff1a91a17e"
  ```
- **Download all missing diagrams across the entire graph**:
  ```powershell
  cd PythonServer
  python -m src.core.extensions.skillsets.KnowledgeGraph.DiagramDownloader
  ```

---

## Step 4: Build Diagram Communities

**Script**: `PythonServer/src/core/extensions/skillsets/KnowledgeGraph/InnovatorDiagramCommunityBuilder.py`

### Purpose
Partitions the knowledge graph by diagram:
- Creates a dedicated `:Community:Dataset` node for every diagram, using an ID such as `Innovator_<Diagram_Name>`.
- Connects the diagram node to the community via `[:AGGREGATES]`.
- Connects **all elements appearing on the diagram** (via `[:diagram_occurrences]`) to that community with `[:AGGREGATES]`.
- Fully idempotent: Re-running uses `MERGE` and will not produce duplicate communities or duplicate relationships.

### How to Run

#### Option A: Via VS Code Debugger
Select **"Python Debugger: Innovator Diagram Community Builder"** in `.vscode/launch.json`:
```json
{
    "name": "Python Debugger: Innovator Diagram Community Builder",
    "type": "debugpy",
    "request": "launch",
    "module": "src.core.extensions.skillsets.KnowledgeGraph.InnovatorDiagramCommunityBuilder",
    "cwd": "${workspaceFolder}/PythonServer",
    "args": [
        "--base_community_name", "Innovator",
        "--repository", "TestModelUAKI",
        "--model-uuid", "76414caa-d57e-41e3-924c-deff1a91a17e"
    ]
}
```

#### Option B: Via Terminal
```powershell
cd PythonServer
python -m src.core.extensions.skillsets.KnowledgeGraph.InnovatorDiagramCommunityBuilder `
    --base_community_name "Innovator" `
    --repository "TestModelUAKI" `
    --model-uuid "76414caa-d57e-41e3-924c-deff1a91a17e"
```

---

## Post-Import Verification Queries in Neo4j

After running all 4 steps, you can execute the following Cypher queries in Neo4j Browser to verify the integrity of the data:

### 1. Verify Ontology & Communities
```cypher
MATCH (o:Ontology {id: "Innovator"})-[:CONTAINS]->(e:EntityOntology)
RETURN o.name AS ontology, count(e) AS total_entities;
```

### 2. Verify Diagrams and Communities
```cypher
MATCH (c:Community)
WHERE toLower(c.id) STARTS WITH "innovator_"
OPTIONAL MATCH (c)-[:AGGREGATES]->(n)
RETURN c.id AS community_id, count(n) AS nodes_in_community
ORDER BY nodes_in_community DESC
LIMIT 10;
```