Dossier Management

This document describes the dossier data model, the Neo4j relationships involved, and the document search cascade implemented in DocumentFetcher.

Dossier Types

Type	Description
Standard dossier	A named collection of documents created by a user or the system. Loaded explicitly into the plan via `DossierLoader`.
Session dossier	A temporary dossier created automatically at session start (id convention: `session-<session_id>`). Holds documents uploaded during the session. Deleted and recreated on each new session.
User dossier	The personal document library of a user, owned by the user's private group in Neo4j (see graph path below).

Neo4j Graph Relationships

User → User Dossier

(u:User {id: user_id})
  -[:MEMBER_OF]->
(g:Group {isUserGroup: true})
  -[:OWNS]->
(d:Dossier)

The isUserGroup: true property distinguishes the personal group (exactly one per user) from other groups.

Dossier → Documents

(d:Dossier)-[:CONTAINS]->(doc:Document)

The primary relationship for dossier–document links is [:CONTAINS]. This is used by all fetching, counting, linking, and reverse-lookup operations in DossiersService.

Dossier Optional Relationships

Relationship	From → To	Purpose
`[:CONTAINS]`	Dossier → Document	Primary dossier–document link
`[:FOLLOWS]`	Dossier → Ontology	Dossier follows a specific ontology
`[:BASED_ON]`	Dossier → DocumentTemplate	Dossier was created from a template
`[:REASONS_WITH]`	Dossier → Document	Dossier uses a reasoning document

Plan Dossier

The active dossier loaded into a conversation plan is stored in Plan.plan_dossier_id. It is set via DossierLoader (skill) or programmatically via plan.set_dossier_id(). Its document summaries are injected into the context by the Contextualizer.

Session Dossier

Stored in Plan.plan_session_dossier_id. Set automatically at session creation. Its document full text is injected into the context by the Contextualizer.

Service API

`DossiersInterface` / `DossiersService`

Method	Description
`get_dossier(dossier_id)`	Returns dossier metadata including document ids (via `[:CONTAINS]`), template ids (via `[:BASED_ON]`), reasoning ids (via `[:REASONS_WITH]`), and followed ontology (via `[:FOLLOWS]`).
`get_all_dossiers()`	Returns all dossiers including optional `follows_ontology_id`, `follows_ontology_name`, `template_id`, and `template_title` when the corresponding relationships exist.
`get_documents_of_dossier(dossier_id, include_details)`	Returns documents linked to a dossier via `[:CONTAINS]`. With `include_details=True`, also returns text, summary, keywords, categories, and timestamps.
`get_ontology_id_for_dossier(dossier_id)`	Returns the ontology ID the dossier follows via `[:FOLLOWS]`, or `None`.
`get_dossiers_for_document(document_id)`	Returns all dossiers that contain the given document via `[:CONTAINS]`.
`get_user_dossier_id(user_id)`	Traverses `(User)-[:MEMBER_OF]->(Group {isUserGroup:true})-[:OWNS]->(Dossier)` and returns the dossier id, or `None` if not found.
`count_documents_in_dossier(dossier_id)`	Returns the number of documents linked via `[:CONTAINS]`.
`link_document_to_dossier(dossier_id, document_ids)`	Creates `[:CONTAINS]` relationships from the dossier to the given documents.
`create_dossier(id, name, description)`	Creates or merges a dossier node.
`delete_dossier(dossier_id)`	Deletes a dossier node and its relationships.
`get_category_usage_stats_in_dossier(dossier_id, ontology_id)`	Returns keyword category usage statistics for documents in the dossier.

The user_id passed to get_user_dossier_id is the JWT-validated effective_user_id returned by get_user_service().get_user_uuid() (see UserService / PYTHONSERVER_JWT_AUTHENTICATION.md) — it is not derived by splitting the session id string.

DocumentFetcher Semantic Search Cascade

DocumentFetcher (skill: PythonServer/src/core/extensions/skillsets/Documents/DocumentFetcher.py) supports a semantic_search argument. When provided without an explicit dossier_id, it runs a 4-step cascade, returning as soon as a scope produces ranked matches.

Parameters

Parameter	Type	Description
`document_ids_filter`	`list`	Optional list of document ids to restrict the result set.
`dossier_id`	`str`	Optional dossier id. When provided, only documents in that dossier are considered.
`semantic_search`	`str`	Optional natural-language query. Triggers reranker + LLM verification (or cascade if no `dossier_id`).
`override_cascade_search`	`bool`	If `True`, skips the cascade and runs a direct reranker search within `dossier_id` (or all documents if not set). No LLM verification in this mode.
`goal`	`Goal`	Optional plan goal. Used by cascade steps 2 (plan dossier) and internally to derive session context.

Cascade Steps (in priority order)

Step	Scope	Source	Missing → behaviour
A	Explicit dossier	`dossier_id` argument	— (cascade skipped, rank immediately)
1	Session dossier	`plan.get_session_dossier_id()`, fallback to `session-<session_id>` from `get_session_id()`	`WARNING` if neither source resolves a dossier
2	Plan dossier	`plan.get_dossier_id()`	`INFO` if no dossier loaded on plan; `INFO` if `goal` is `None`
3	User dossier	`DossiersService.get_user_dossier_id(user_id)`	`WARNING` if no user dossier found
4	All documents	Full repository (fallback)	Always executes

Steps 1 and 3 attempt resolution regardless of whether goal is None, using session/user context from src/config/session.py. Step 2 requires goal to be set.

DossierType enum

The DocumentFetcher result includes a from_dossiers field — a list of dicts with id, type, name, and description identifying the dossier(s) from which results originate. The type field uses values from the DossierType enum defined at the top of DocumentFetcher.py:

Value	When set	`from_dossiers` content
`SPECIFIED_DOSSIER`	Caller passed an explicit `dossier_id` argument	single entry with the provided dossier id
`SESSION_DOSSIER`	Cascade resolved at step 1 — results from the session upload dossier	single entry with session dossier id
`LOADED_DOSSIER`	Cascade resolved at step 2 — results from the dossier currently loaded into the plan	single entry with plan dossier id
`USER_DOSSIER`	Cascade resolved at step 3 — results from the user's personal dossier	single entry with user dossier id
`GLOBAL_DOCUMENTS`	No `dossier_id` given and no semantic search, or cascade fell through to step 4	list of dossiers owning the returned documents (looked up via `get_dossiers_for_document()`)

Return structure

{
    "success": True,
    "output": {doc_id: {"docid", "name", "summary", "created", "updated"}},
    "from_dossiers": [{"id": dossier_id, "type": DossierType, "name": ..., "description": ...}],
    "instructions": "..."
}

Ranking and verification logic

Phase 1: Reranker — Documents with a summary are passed to the reranker (search_documents_by_semantic). Documents without a summary cannot be ranked and are excluded from this phase.

Phase 2: LLM Verification — For cascade steps A (explicit dossier), 1 (session), 2 (plan), and 3 (user), if the reranker returns at least one result, the LLM is called to validate whether those results genuinely match the user's request. The LLM uses a strict binary prompt with positive and negative examples, and receives the document names, summaries, and dossier context (dossier ID and type). The LLM returns a comma-separated list of confirmed document IDs, or null if none match. Only LLM-confirmed documents are returned; LLM rejection causes the cascade to continue to the next step. On LLM failure, the function falls back to returning the ranked docs unchanged (fail-open).

Step 4 (fallback) — At the final step 4 (all documents), reranking is performed but no LLM verification occurs, as the downstream agent (AgentToolPreparer) will provide additional evaluation. Documents without summaries are merged back in only at this step.

Cascade resolution — A step is considered to have resolved only when: 1. The reranker returns at least one result, and 2. The LLM confirms at least one of those results (or step 4 is reached, which has no LLM gate)

No-summary-only dossiers and LLM-rejected results do not stop the cascade prematurely.

Private helpers

Function	Purpose
`_fetch_raw_documents(document_ids_filter, dossier_id)`	Fetches and normalises document metadata applying id/dossier filters. Uses `get_document_ids_for_dossier()` then `get_documents()` when a `dossier_id` is given.
`_fetch_docs_for_dossier(dossier_id)`	Thin wrapper around `_fetch_raw_documents` for a single dossier; returns `{}` and logs `WARNING` on error.
`_rank_docs(candidate_docs, query)`	Calls `search_documents_by_semantic`; returns only ranker-selected docs (no-summary docs excluded from return value so callers can detect "no match").
`_llm_verify_docs(ranked_docs, query, dossier_id, dossier_type)`	Asks the LLM to confirm which reranked documents genuinely match the query. Returns LLM-confirmed docs or `{}` if LLM rejects all. Falls back to `ranked_docs` unchanged if the LLM call fails.
`_semantic_search_cascade(all_documents, query, explicit_dossier_id, goal)`	Orchestrates the full cascade described above, applying reranker + LLM verification for steps A, 1, 2, 3 and reranker only for step 4. Returns `(docs_dict, from_dossiers_list, DossierType)`.