Category Search Algorithm Documentation

Overview

The thalabus_query() method in RagService implements a sophisticated query optimization system that intelligently delegates category-only expressions to the OntologyService for efficient Cypher-level processing. The system analyzes queries, identifies category-only subtrees, and executes them as single optimized Cypher queries instead of splitting and combining results in Python.

Query Optimization Through Internal Representation

The @ Delegation Mechanism

The system uses an internal representation where category-only subtrees are marked with @ symbols. This is purely internal and not exposed to users.

Example: - User input: "asylum seeker" AND COUNTRY_OF_ORIGIN=IQ - Internal representation: "asylum seeker" AND @(COUNTRY_OF_ORIGIN=IQ) - The @ marks that COUNTRY_OF_ORIGIN=IQ can be delegated to categories_query() - The semantic search part remains in Python for processing

Complex example: - User input: (DEPORTATION_DESTINATION_COUNTRY=IQ AND NOT DISABILITY_TYPE=TRAUMA) OR COUNTRY_OF_ORIGIN=SY - Internal representation: @((DEPORTATION_DESTINATION_COUNTRY=IQ AND NOT DISABILITY_TYPE=TRAUMA) OR COUNTRY_OF_ORIGIN=SY) - The entire expression is category-only, so it's marked with @ and delegated as a single Cypher query

Optimization Process

The query optimization happens in three phases:

  1. Parsing Phase:
  2. Tokenize the input query into tokens

  3. Build an Abstract Syntax Tree (AST) representing the logical structure

  4. Marking Phase (_mark_category_only_subtrees):

  5. Recursively traverse the AST from innermost to outermost expressions
  6. For each subtree, check if all categories exist in the ontology using is_categories_only_query()
  7. Mark category-only nodes with is_category_only=True and store the serialized query in category_query field

  8. Non-category parts (semantic/text search) remain unmarked

  9. Evaluation Phase (_evaluate_thalabus_ast):

  10. Check if node is marked as category-only
  11. If yes: delegate entire subtree to ontology_service.categories_query() for efficient Cypher execution
  12. If no: recursively evaluate children and combine results using set operations (union for OR, intersection for AND)

How It Works

Input Format

The method accepts a boolean expression query with the following syntax:

Examples

  1. Category only
  2. IS_DEAD

  3. DUBLIN_ARTICLE=ART_13,ART_25_2

  4. Semantic search with category

  5. "Bob the Cat" AND SPECIES=CAT

    • Semantically searches for a cat named Bob, restricted to documents where SPECIES is CAT.

  6. Exact text search (case-insensitive)

  7. 'bob'

    • Finds documents containing "bob", "Bob", "BOB", etc.

  8. Exact text search (case-sensitive)

  9. @'Bob'

    • Finds documents containing exactly "Bob" (case-sensitive).

  10. NOT operator example

  11. NOT COUNTRY=IQ
    • Finds documents that are NOT in the IQ category.

  12. "Bob the Cat" AND NOT SPECIES=DOG

    • Semantically searches for Bob the Cat, but excludes documents where the species is DOG.

  13. Combined complex example

  14. ("Bob the Cat" OR 'bob') AND (SPECIES=CAT OR SPECIES=DOG) AND NOT IS_DEAD

    • Finds documents semantically related to "Bob the Cat" or containing the exact text "bob" (any case), where the species is either CAT or DOG, but excludes documents where IS_DEAD is true.

  15. General-information semantic query

  16. "steps and procedures for returning a rented car from a car rental company"
    • Represents a general-information request as a valid semantic query. AgentPromptAugmenter uses this form rather than emitting TRUE.

Common Mistakes

1. Missing Boolean Operator Between Expressions

❌ Incorrect:

CATEGORY1 NOT CATEGORY2

✅ Correct:

CATEGORY1 AND NOT CATEGORY2

Error message: Unexpected tokens after parsing: ['NOT', ('CATEGORY', 'CATEGORY2')]

Explanation: The NOT operator must be preceded by AND or OR. You cannot place two expressions next to each other without a boolean operator.

2. Standalone NOT Operator

❌ Incorrect:

NOT COUNTRY=IQ

✅ Correct:

TRUE AND NOT COUNTRY=IQ

or as part of a larger expression:

COUNTRY=SY AND NOT COUNTRY=IQ

Error message: NOT operator cannot be used standalone. Use it with AND: "X AND NOT Y"

Explanation: While NOT can be parsed, it must be combined with another expression using AND or OR.

3. Missing Quotes for Multi-Word Searches

❌ Incorrect:

Bob the Cat AND SPECIES=CAT

✅ Correct:

"Bob the Cat" AND SPECIES=CAT

Error message: Tokenization errors or unexpected parsing results.

Explanation: Multi-word phrases must be enclosed in double quotes for semantic search or single quotes for exact text search.

Algorithm Steps

Step 1: Tokenization (_tokenize_thalabus_query)

Step 2: Parsing (_parse_thalabus_expression, _parse_thalabus_and_expression, _parse_thalabus_not_expression, _parse_thalabus_primary)

Step 3: Marking Category-Only Subtrees (_mark_category_only_subtrees)

Step 4: Evaluation (_evaluate_thalabus_ast)

Two independent parser implementations (found during documentation review): The query language described in this document is parsed twice, by two separate implementations that must be kept in sync when the grammar changes: - RagService (_tokenize_thalabus_query, _parse_thalabus_expression, _mark_category_only_subtrees, _evaluate_thalabus_ast) — the primary entry point, documented in this file. - OntologyService.categories_query() uses its own tokenizer/parser (_tokenize_query, _parse_expression, _generate_cypher_from_ast) to turn a category-only subtree directly into a Cypher WHERE clause. This is the delegation target described in the "@ Delegation Mechanism" section above. A grammar change (new operator, new quoting rule, etc.) must be applied to both parsers.

Neo4j Graph Pattern

For category-only queries, the algorithm generates optimized Cypher queries that search for this pattern in Neo4j:

(d:Document)-[:FOLLOWS]->(o:Ontology)-[:CONTAINS]->(kc:KeywordCategory)
with (d)-[:HAS_CATEGORY]->(c:Category)-[:OF_CATEGORY]->(kc)

Example

Input query:

IS_DEAD OR (DUBLIN_ARTICLE=ART_13,ART_25_2 AND COUNTRY=IQ)

Internal representation after marking:

@(IS_DEAD OR (DUBLIN_ARTICLE=ART_13,ART_25_2 AND COUNTRY=IQ))

Execution: - The entire expression is marked as category-only - It's delegated to categories_query() as a single Cypher query - The Cypher engine handles all boolean logic and category filtering efficiently

Efficiency Benefits

  1. Cypher-level Optimization: Complex boolean expressions are executed as single Cypher queries, leveraging Neo4j's query optimizer
  2. Reduced Network Roundtrips: Category-only queries execute in one database roundtrip instead of multiple
  3. Better Index Usage: Neo4j can apply indexes across the entire boolean expression
  4. Intelligent Splitting: Mixed queries (semantic + category) are split optimally - only the non-category parts remain in Python
  5. Lazy Evaluation: NOT operations only fetch all document IDs when needed

Key Design Decisions

  1. Internal Representation: The @ notation is purely internal and not part of the user-facing query language
  2. Recursive Marking: Category-only detection works from innermost to outermost, allowing complex nested expressions to be optimized
  3. Validation Cache: The is_categories_only_query() method validates category existence before marking
  4. Set Operations: Python-side evaluation uses set operations (intersection, union, complement) for efficient result combination
  5. Graceful Degradation: If a category doesn't exist, the subtree is marked as non-category, and evaluation falls back to Python processing

Usage Example

from src.services.rag_service import get_rag_service

service = get_rag_service()

# Simple category search (fully optimized)
results, _ = service.thalabus_query("Juris", "COUNTRY_OF_ORIGIN=IQ")
# Log shows: Delegating category-only subtree to categories_query(): COUNTRY_OF_ORIGIN=IQ

# Complex category search (fully optimized)
results, _ = service.thalabus_query(
    "Juris", 
    "(COUNTRY_OF_ORIGIN=IQ OR COUNTRY_OF_ORIGIN=SY) AND NOT IS_DEAD"
)
# Log shows: Delegating category-only subtree to categories_query(): ((COUNTRY_OF_ORIGIN=IQ OR COUNTRY_OF_ORIGIN=SY) AND NOT (IS_DEAD))

# Mixed search (partially optimized)
results, _ = service.thalabus_query(
    "Juris",
    '"refugee crisis" AND COUNTRY_OF_ORIGIN=IQ'
)
# Logs show:
# - Semantic search "refugee crisis" processed separately
# - Delegating category-only subtree to categories_query(): COUNTRY_OF_ORIGIN=IQ
# - Results combined with AND operation

Testing

Run the test script to verify functionality:

python src/services/rag_service.py

Look for log messages showing: - RagService._mark_category_only_subtrees: Marked as category-only: — Subtrees that were optimized - RagService._mark_category_only_subtrees: Not category-only: — Subtrees that require Python evaluation - RagService._evaluate_thalabus_ast: Delegating category-only subtree to categories_query(): — Actual delegation points - RagService.thalabus_query: Optimized query representation: — The full query with internal @ marks

Error Handling

Automatic Reranking (Dec 2025)

The thalabus_query() method now supports automatic reranking of results:

def thalabus_query(
    self,
    query: str,
    ontology_id: Optional[str] = None,
    subset_of_datasetids: Optional[List[str]] = None,
    automatically_rerank: bool = False,
    create_dataset: Optional[Union[bool, str]] = None,
    session_id: Optional[str] = None,
    userid: Optional[str] = None
) -> Tuple[List[dict], Optional[List[str]]]:
    # Returns (document_list, dataset_ids)
    # dataset_ids is a list with single dataset ID if created, None otherwise

Parameters

How Reranking Works

When automatically_rerank=True:

  1. Semantic/text search executes normally and returns initial results
  2. Statistical filtering (standing out results) is disabled when reranking is enabled
  3. Reranking service fetches chapters and scores them in parallel
  4. Results are re-sorted by reranking scores (descending)

Example with Reranking

results, _ = service.thalabus_query('"cleanup with the mop"', automatically_rerank=True)
# Returns tuple: ([docid-543 (score: 13.03), docid-544 (score: 9.80), ...], None)

See RERANKING_ARCHITECTURE_DEC2025.md for complete details.

Dataset Creation (Jan 2026)

The thalabus_query() method now supports automatic dataset creation from query results:

Parameters

How Dataset Creation Works

When create_dataset is set:

  1. Query executes normally and returns fragment results (chapters, paragraphs, summaries)
  2. If subset_of_datasetids is provided, only fragments from documents in those parent datasets are included
  3. Dataset name is determined:
  4. If create_dataset=True: Auto-generated with prefix "query_" (e.g., "query_001")
  5. If create_dataset="my_dataset": Uses the provided name
  6. Dataset is created with:
  7. All fragment IDs from query results (preserves actual matched fragments)
  8. Fragment scores as weights (truncated to 5 decimal places)
  9. Metadata: query string, ontology_id, item counts, fragment type breakdown
  10. Dataset ID is returned in a list as the second tuple element

Important: Datasets now contain the actual fragments that matched the query, not aggregated documents. This means: - Semantic searches ("text") → mostly c:Chapter and p:Paragraph nodes - Category searches (CATEGORY) → d:Document nodes (categories are document-level) - Text searches ('text') → mix of c:Chapter nodes and keyword fragments - Mixed queries → combination of all fragment types

Examples with Dataset Creation

# Auto-generate dataset name
results, dataset_ids = service.thalabus_query(
    '"refugee crisis" AND COUNTRY_OF_ORIGIN=IQ',
    ontology_id="Juris",
    create_dataset=True,
    session_id="user123-session456"
)
# Returns: (document_list, ["dataset-query_001"])

# Use custom dataset name
results, dataset_ids = service.thalabus_query(
    'MEDICAL_CONDITION AND "diabetes"',
    create_dataset="medical_diabetes_cases",
    session_id="user123-session456",
    userid="user123"
)
# Returns: (document_list, ["dataset-medical_diabetes_cases"])

# No dataset creation (default behavior)
results, dataset_ids = service.thalabus_query('"cleanup with the mop"')
# Returns: (document_list, None)

# Restrict search to specific parent datasets (multi-parent OR logic)
results, dataset_ids = service.thalabus_query(
    '"asylum seeker" AND COUNTRY=IQ',
    ontology_id="Juris",
    subset_of_datasetids=["dataset-legal-cases", "dataset-reference-docs"],
    create_dataset="filtered_asylum_cases",
    session_id="user123-session456"
)
# Returns: (document_list filtered to only docs in the parent datasets, ["dataset-filtered_asylum_cases"])
# The created dataset will be a child of both parent datasets

Dataset Structure

Created datasets include: - Fragments: All fragment IDs from query results (can be Documents, Chapters, Paragraphs, or Summaries) - d:Document nodes for document-level matches (e.g., category filters) - c:Chapter nodes for chapter-level matches (fragmentid: "docid-c1", "docid-c2", ...) - p:Paragraph nodes for paragraph-level matches (fragmentid: "docid-p1", "docid-p2", ...) - s:Summary nodes for summary-level matches (fragmentid: "docid-s") - Weights: Fragment scores (0.0-1.0, truncated to 5 decimals) - Metadata: - query: Original query string - ontology_id: Ontology used in search - item_count: Number of fragments in dataset - fragment_types: Breakdown of fragment types (e.g., {"chapters": 50, "paragraphs": 120, "documents": 10}) - result_count: Number of aggregated document results from query - creation: Set to "thalabus_query"

Fragment Types in Datasets

The type of nodes stored in the dataset depends on the query type. This table explains when each fragment type is passed to the dataset:

Query Type Fragment Types in Dataset Reason
Category-only queries
(e.g., CATEGORY=VALUE)		 d:Document nodes Categories are document-level metadata. The category filter returns entire documents.
Semantic search queries
(e.g., "refugee crisis")		 c:Chapter, p:Paragraph, s:Summary nodes Semantic search uses vector similarity on chapter/paragraph/summary embeddings. Returns the specific fragments that match.
Text search queries
(e.g., 'specific text')		 c:Chapter nodes (primarily) Fulltext index searches chapter text and keywords. Returns chapters containing the text.
Mixed queries
(e.g., "asylum" AND COUNTRY=IQ)		 Mixed types
p:Paragraph + d:Document		 AND combines fragments from both branches. Left branch returns paragraphs, right branch returns documents.
OR queries
(e.g., "text" OR CATEGORY)		 Mixed types
Union of all fragment types		 OR unions all fragments from all branches.

User presentation: While datasets store fragments, the thalabus_query() return value aggregates fragments to documents for user-friendly results.