# Datasets

This document is the consolidated reference for Thalabus datasets across Python,
Neo4j, C#, and Razor. It covers dataset creation and propagation, graph
membership, relationship visibility, reranking, UI rendering, and dataset URI
navigation.

## Dataset Model

Datasets are structured data artifacts that move from Python goals to C# and
Razor through `goal_datasets` and `Datasets`.

A dataset may contain documents, chapters, paragraphs, summaries, knowledge
graph entities, dataset objects, or other supported graph nodes. A `:Community`
node is also labeled `:Dataset`, but not every dataset is a community. Communities
are permanent datasets representing an ontology entity namespace; ordinary
permanent datasets remain valid without community semantics.

Dataset IDs are canonical and may be session-qualified:

```text
luca-s-2#maktumin_statelessness_certificate_1
```

Session-scoped datasets use `session_id#dataset_name`. Permanent datasets use a
bare dataset ID and are not attached to a session. The UI must preserve the
canonical ID when querying fragments. A short name is only a fallback when no
canonical ID is available.

### Community identity

Every `:Community` node is also a `:Dataset` node, but ordinary datasets are not
communities. Communities are always permanent and are never session-qualified.
The Community `id` is the only community identifier and must be used for lookup,
navigation, membership, and rename operations. Community nodes must not carry a
separate `name` property.

For example, `DefaultCommunity_FictionalStories` is the permanent `id` of the
default community for the `FictionalStories` ontology. This rule does not change
the general dataset contract above: ordinary session-scoped datasets may still
use `session_id#dataset_name`.

The `/datasets/communities` rename operation changes the Community `id`. It must
not read or write a separate Community `name` attribute.

## Dataset Lifecycle

The conceptual flow is:

```text
RAG fragment candidates
  -> AgentReranker reranks and materializes a dataset
  -> Consolidators propagate and aggregate dataset references
  -> Goal stores goal_datasets
  -> Session projection copies datasets to ContainerMessage.Datasets
  -> Chat, Table View, and ChatLineSources render the dataset
```

### Control plane and data plane

The control plane contains goals that reason, decide, or route execution, such
as User, Strategist, and Planner. The data plane contains goals that retrieve,
score, rank, and persist evidence, such as RAG, Reranker, and Consolidators.
Datasets are data-plane artifacts.

### RAG retrieval

`AgentRAG` and related RAG goals retrieve document or knowledge graph fragment
candidates and compute relevance scores. At this stage the fragments are
execution metadata, not yet reusable datasets.

### Dataset genesis: AgentReranker

`AgentReranker` is the first goal that materializes a dataset. It:

1. consumes RAG fragment candidates;
2. applies reranking logic;
3. selects and stabilizes a result set; and
4. persists the result as a dataset, commonly with a `_reranker` suffix.

The persisted relationship `weight` is the reranker score. The incoming
retrieval score is retained only in transient reranking results as
`original_score`; it is not used to rescale the reranker score.

### Upward propagation: Consolidators

Consolidators do not create datasets from scratch. They accept datasets from
child goals, merge summaries, provenance, and references, and attach dataset
identifiers to their own results. They propagate knowledge about datasets; the
dataset remains the persisted artifact.

`AgentPlanner` coordinates execution and decides that retrieval or reranking is
needed, but does not create datasets, rerank fragments, or consolidate evidence.

## Dataset Graph Relationships

### Unified membership: `:AGGREGATES`

Both ordinary datasets and communities point to their member nodes through a
single unified relationship type, using ArchiMate/UML aggregation semantics
(non-owning, shared, whole-part):

```cypher
(d:Dataset)-[r:AGGREGATES]->(n)
```

The target `n` may be a `Document`, `Chapter`, `Paragraph`, `Entity`,
`DatasetObject`, or another supported graph node. The relationship may contain
the following optional/nullable properties, regardless of whether the source
node is an ordinary `Dataset` or a `Community` (a `:Community` node is also
labeled `:Dataset` — see below). There is no schema branching by source node
type; only presence/absence of these optional properties, read via
`coalesce(...)` exactly like `hidden` below:

| Property | Type | Meaning |
|---|---|---|
| `weight` | float | Numeric relevance or ranking value |
| `hidden` | boolean or null | Dataset-specific visibility flag |
| `element_ids` | list of strings | HTML element IDs highlighted for this dataset relationship |

Community-sourced `:AGGREGATES` edges (`(c:Community)-[:AGGREGATES]->(e:Entity)`)
never set `weight`, `hidden`, or `element_ids` — they are simply absent/null,
same as they may already be absent for some ordinary dataset edges today.

Dataset queries may traverse up to three hops:

```cypher
(d:Dataset)-[:AGGREGATES*1..3]->(n)
```

The first hop reaches direct members. Additional hops can reach source nodes
linked from generated `DatasetObject` nodes. Dataset objects have an `id` and
may carry input or provenance properties such as `source_node_id`,
`a_node_id`, and `b_node_id`.

Dataset lineage is represented separately:

```cypher
(child:Dataset)-[:SUBSET_OF]->(parent:Dataset)
```

The ontology path to a community's entities is:

```cypher
(o:Ontology)-[:DEFINES]->(c:Community)-[:AGGREGATES]->(e:Entity)
```

#### Migration history

`:AGGREGATES` unifies what used to be two separate relationship types:
`(Dataset)-[:ASSOCIATED_WITH]->(n)` and `(Community)-[:CONTAINS]->(Entity)`.
The one-time idempotent rename is documented at
[Neo4jORM/migrations/02_rename_dataset_community_relationships_to_aggregates.cypher](../../Neo4jORM/migrations/02_rename_dataset_community_relationships_to_aggregates.cypher)
(a reviewable reference copy); the statements that actually execute at
utility-container boot are embedded in
`PythonServer/src/core/base/Maintenance.py`
(`migrate_dataset_community_relationships_to_aggregates()`), called once from
`knowledge_graph_maintenance_thread()`.

**Note:** `:CONTAINS` remains in active use elsewhere for the *unrelated*
`(Dossier)-[:CONTAINS]->(Document)` ownership relationship (renamed from
`:OWNS` by `acl_migration.cypher` Section 1). Do not confuse the two — that
`:CONTAINS` usage has nothing to do with dataset/community membership.

**Status: complete.** All C# dataset/community readers and writers
(`DatasetService.cs`, `Neo4jDatasetRepository.cs`, `Neo4jOntologies.cs`) and all
Python callers (`dataset_service.py`, `rag_service.py`, `ontology_service.py`,
and the `KnowledgeGraph`/`Documents` skillsets: `KnowledgeGraphEditor.py`,
`KnowledgeGraphReader.py`, `KnowledgeGraphConsolidator.py`,
`InnovatorDiagramCommunityBuilder.py`, `ProsgresInnovatorKnowledgeGraphImport.py`,
`KgFromDocument.py`) read and write `:AGGREGATES` exclusively. The live Neo4j
instance was verified to hold zero `(Dataset)-[:ASSOCIATED_WITH]->()` and zero
`(:Community)-[:CONTAINS]->()` edges anywhere in the graph; all prior instances
were converted to `:AGGREGATES` with their `weight`/`hidden`/`element_ids`
properties preserved.

## The `hidden` Relationship Property

`hidden` belongs to the `:AGGREGATES` relationship between a dataset and a
member node. It is not a property of the `Dataset`, `Document`, `Entity`, or
other target node.

It is dataset-specific: the same node can be visible in one dataset and hidden
in another without changing the node or its other dataset relationships.

| Value | Meaning | UI result |
|---|---|---|
| `null` | Not set; default state | Visible |
| `false` | Explicitly visible | Visible |
| `true` | Explicitly hidden in this dataset | Hidden by default |

When `hidden` is absent, the C# repository reads it using
`coalesce(r.hidden, false)`, so the fragment is visible by default.

### Reading visibility

The C# dataset repository reads relationship visibility as follows:

```cypher
MATCH (d:Dataset {id: $datasetId})-[r:AGGREGATES]->(n)
RETURN n.id AS nodeId,
       labels(n) AS nodeLabels,
       r.weight AS weight,
       r.element_ids AS elementIds,
       coalesce(r.hidden, false) AS hidden
ORDER BY r.weight DESC
```

The result is mapped to `INeo4jDatasetFragment.Hidden`.

### Setting visibility

Hide selected relationships:

```cypher
MATCH (d:Dataset {id: $datasetId})-[r:AGGREGATES]->(n)
WHERE n.id IN $nodeIds
SET r.hidden = true
```

Hide every relationship except selected nodes:

```cypher
MATCH (d:Dataset {id: $datasetId})-[r:AGGREGATES]->(n)
SET r.hidden = CASE
    WHEN n.id IN $keep_node_ids THEN null
    ELSE true
END
```

The Python service exposes `set_dataset_relationships_hidden()` for selected
node IDs and `hide_dataset_relationships_except()` for the keep-list behavior.
`upsert_dataset_fragments()` accepts `hidden: Optional[bool]`; its default
`None` leaves the relationship without an explicit visibility value.

### Read-only datasets

Permanent datasets may be marked with `readonly = true`. Python mutations that
change dataset contents or `AGGREGATES` relationship properties raise
`DatasetReadOnlyError` when targeting such a dataset. Session-scoped datasets
cannot be marked read-only. Dataset creation, duplication, set operations,
lifecycle management, deletion, and session cleanup are not blocked by this
content-mutation guard.

### Explicit hiding by `AgentAssistant`

`AgentAssistant` explicitly changes relationship visibility after it has
collected or cloned datasets and produced the final summary. Its
`_restrict_dataset_visibility_to_summary_references()` method extracts the
`doc://`, `kg://`, and `inno://` node references present in the summary, then
calls:

```python
dataset_service.hide_dataset_relationships_except(dataset_id, node_ids)
```

That method sets `r.hidden` to `null` for nodes in `node_ids` and to `true`
for every other node in the dataset. Consequently, a KG entity can have a
ranking above `0.5` and still be hidden when its `kg://` reference is not
present in the final assistant summary. This is an explicit visibility
operation; it is not an implicit consequence of ranking or reranking.

The operation applies to the assistant's cloned output datasets. It does not
normally originate in `AgentReranker`, whose dataset creation persists
`AGGREGATES.weight` but does not set relationship `hidden`.

To inspect visibility for one dataset:

```cypher
MATCH (d:Dataset {id: $datasetId})-[r:AGGREGATES]->(n)
RETURN
    count(CASE WHEN r.hidden = true THEN 1 END) AS hidden_count,
    count(CASE WHEN r.hidden IS NULL OR r.hidden = false THEN 1 END) AS visible_count
```

## RAG and Reranking

The relevant score meanings are distinct:

| Stage | Producer | Meaning |
|---|---|---|
| Retrieval | `RagService.thalabus_query()` or vector search | Embedding similarity, optionally combined across semantic branches |
| Reranking | External `/score` API through `RerankingService` | Pairwise relevance score for the query and fragment text |
| Dataset | `AgentReranker` | Reranker score persisted as `AGGREGATES.weight` |

The external reranker score is not a calibrated replacement for the embedding
score. A fragment can rank first while having a small absolute reranker score;
comparisons should use the score from the same stage.

`AgentReranker` separates document and `kg://` fragments for reranking. KG
fragments are loaded through the ontology service and reranked independently.
The implementation guarantees a minimum KG output when the service filters too
aggressively, then persists the selected fragments and their scores as dataset
relationships.

The reranker does not normally set `AGGREGATES.hidden`. A high or low
ranking score and relationship visibility are independent properties. A KG
fragment with a score above the relevance threshold can still be absent from
the default UI only when its relationship has `hidden = true` or when another
layer excludes it.

## Dataset Rendering

### Dataset contents list

`ThalabusWeb/RazorComponents/Datasets/DatasetContentsList.razor` renders the
contents of one dataset in an expandable curtain.

Document fragments (`FragmentResourceType.Doc`) are grouped by:

```csharp
DocumentManagementService.GetDocumentId(fragment.Path)
```

Within a document group, fragments are ordered by descending ranking and the
document heading shows the highest ranking. A document with only one visible
fragment is rendered as a single top-level row.

Non-document resources remain top-level entries. This includes:

- knowledge graph resources, rendered through `KgEntityLink`;
- HTTP and HTTPS source links; and
- other supported resource types.

The component first filters relationship visibility:

```csharp
fragment => showAllFragments || !fragment.Hidden
```

The default view removes fragments whose `AGGREGATES.hidden` value is
true. `ShowAllFragments` includes hidden relationships, but does not disable
relevance filtering.

After visibility filtering, the component applies a 20% relevance threshold:

- multi-fragment document groups retain fragments with ranking at least `0.2`;
- a single-fragment document bypasses that group threshold; and
- every non-document fragment needs ranking at least `0.2`.

Thus `hidden` and ranking are independent. A fragment with ranking above `0.5`
can still be omitted from the default list if its relationship is hidden.

Top-level groups are ordered by descending ranking and then by key using
case-insensitive ordering. Ranking is shown as a percentage title and up to
five circle indicators in 20% steps.

Document text is loaded separately, preferring `EnhancedText` and falling back
to `Text`. KG entities are resolved through `IOntologyService`; lookup failure
falls back locally without changing relationship visibility.

### DatasetCard parameters

`ThalabusWeb/RazorComponents/Datasets/DatasetCard.razor` is a reusable card
whose layout and behavior are controlled by its parameters:

| Parameter | Type and default | Effect |
|---|---|---|
| `Dataset` | `INeo4jDataset` (required) | Supplies the dataset name, description, domain, ID, permanence, and object count. |
| `IsSelected` | `bool` | Adds the `selected` state used to highlight the active card. |
| `OnDatasetSelected` | `EventCallback<INeo4jDataset>` | Runs when the card or compact expand/collapse control is clicked. |
| `OnDatasetDeleted` | `EventCallback<INeo4jDataset>` | Runs after the delete confirmation is accepted. |
| `IsCompact` | `bool`, `false` | Enables the compact layout, dataset link, expand/collapse control, and automatic count loading. |
| `ShowCardDetails` | `bool`, `true` | Shows or hides the card header, description, domain, badges, and full-mode actions. |
| `ShowFragments` | `bool`, `false` | Requests dataset fragments; it is effective only when `IsCompact` is `true`. |
| `ShowMetadata` | `bool`, `false` | In compact mode, controls whether the description and domain are shown. |
| `DocumentCount` | `int?`, `null` | Shows a document badge when greater than zero; in compact mode, a missing value is loaded automatically. |
| `KnowledgeGraphNodeCount` | `int?`, `null` | Shows a KG badge when greater than zero; in compact mode, a missing value is loaded automatically. |

The card passes fragment-related behavior to
`DatasetContentsList.razor`. The effective fragments flag is
`IsCompact && ShowFragments`, so a full dataset card cannot expand and load
fragments accidentally. The `ShowAllFragments` state is internal to the card;
the full dataset-card view does not expose a visible toggle for it.

Current configurations are:

| Usage | Configuration | Behavior |
|---|---|---|
| `/datasets` | `IsCompact=false` | Full card with description, domain, actions, and selection highlight; fragments remain hidden. |
| Chat right panel | `IsCompact=true` | Compact card; selecting it can expand the fragment curtain. |
| Knowledge-graph node details | `IsCompact=true`, `ShowFragments=false`, `ShowMetadata=true` | Compact informational card without fragments, with description and domain visible. |

### Dataset-only and source rendering

A dataset-only assistant response is valid when it has no prose summary but
returns a structured dataset, for example through `ds://<dataset_id>` in
`optional_rag_fragments_used`.

| View | Datasets | Rendering |
|---|---:|---|
| Chat | yes | Dataset result or summary with dataset sources |
| Table | yes | Dataset component in the Sources column |
| ChatLineSources | yes | Structured dataset source tabs after document and KG links |

## Dataset URI Navigation

The `ds://<dataset_id>` URI identifies a dataset in assistant summaries and
chat source references.

`ChatLineSources` recognizes structured datasets carried in
`ContainerMessage.Datasets` and `ds://` values carried in RAG fragments. It
merges and deduplicates both forms, then displays dataset source tabs after
document and knowledge graph links.

The adjacent external-link action opens:

```text
/datasets/{Uri.EscapeDataString(dataset_id)}
```

URL encoding is required so temporary IDs containing `#` remain a path segment
instead of becoming a browser fragment. The `/datasets/{DatasetId}` route
preselects the dataset. `DatasetSource` resolves only datasets visible to the
current user and renders the dataset name, description, domain, object count,
and ID.

## Operational Checks

When a dataset appears to omit a document or KG entity, inspect the relationship
rather than only the node or its ranking:

```cypher
MATCH (d:Dataset {id: $datasetId})-[r:AGGREGATES]->(n)
RETURN n.id, labels(n), r.weight, r.hidden, r.element_ids
ORDER BY r.weight DESC
```

Interpret the result as follows:

- `r.hidden = true`: the default UI intentionally suppresses the member;
- `r.hidden IS NULL` or `r.hidden = false`: visibility is not the cause;
- low `r.weight`: check the applicable 20% UI threshold and the upstream
  retrieval/reranking stage;
- no relationship: the node is not a member of that dataset.

For the default UI, selecting **Show All Dataset Fragments** tests only the
visibility condition. It does not bypass ranking thresholds.