Container Management in Thalabus

This document describes how Docker containers are managed in the Thalabus solution: the key classes and interfaces involved, their responsibilities, and the two main activation sequences (standard container activation, and pre-loaded container activation).

---

Table of Contents

1. Architecture Overview
2. Classes and Interfaces Reference
3. Services
4. Container Data Classes
5. Infrastructure Interfaces (ThalabusInterfaces)
6. Supporting Data Structures
7. Container Activation Sequence (Standard Path)
8. Pre-Loaded Container Activation Sequence
9. Startup Orchestration (Program.cs)

---

1. Architecture Overview

The container management layer sits between the .NET application logic and the Docker Engine. It manages:

• Python AI containers – short-lived Docker containers running a Python Flask process that executes skillsets on behalf of user sessions.
• Infrastructure containers – long-lived service containers for Neo4j, Elasticsearch, and Kibana, started once at platform boot.
• A utilities container – a dedicated Python container that acts as an always-on helper service, providing plan configuration, translation, maintenance triggers, and other shared utilities.
• A pre-loaded container pool – a set of warm, idle AI containers kept ready to eliminate cold-start latency when a new session begins.

All container operations go through Docker.DotNet, the official .NET Docker client library.

graph TB
    subgraph NET [".NET Application (ThalabusWeb)"]
        direction LR
        CS["ContainerService (singleton)"]
        CP["ContainerPool (background worker)"]
        CS -- owns --> CP
    end

    subgraph DOCKER ["Docker Engine  ·  shared bridge network"]
        direction LR

        subgraph POOL ["Pre-Loaded Pool (warm — Flask running)"]
            direction TB
            P1["preload-xxxx1 (Flask ready)"]
            P2["preload-xxxx2 (Flask ready)"]
        end

        subgraph SESS ["AI Session Containers (per session)"]
            direction TB
            S1["Session A  (active)"]
            S2["Session B  (active)"]
        end

        subgraph UTIL ["Utilities Container (always-on)"]
            PY["Python utilities (Flask · shared services)"]
        end

        subgraph INFRA ["Infrastructure Containers (long-lived)"]
            direction TB
            NEO["Neo4j  :7474 / :7687"]
            ES["Elasticsearch  :9200"]
            KIB["Kibana  :5601"]
        end
    end

    CS -- "Docker.DotNet start / stop / exec" --> DOCKER
    CS -- "manages" --> INFRA
    CS -- "starts via ContainerUtilities" --> UTIL
    CS -- "creates on demand" --> SESS
    CP -- "pre-fills (background)" --> POOL
    CP -- "TryAcquire → promotes to session" --> SESS

---

2. Classes and Interfaces Reference

Services

ContainerService (ThalabusCore/Services/ContainerService.cs)

The central singleton service that orchestrates all Docker container operations across the platform. Registered as IContainerService in the DI container.

Key responsibilities: - Singleton access – exposes itself via ContainerService.Singleton for non-DI access throughout the codebase. - Environment variables – loads, saves, and resolves EnvironmentVariables (from Configuration/EnvironmentVariables.json) and EnvironmentParameters (runtime key-value pairs injected at startup). Supports {{placeholder}} substitution in variable values. - LLM settings – loads and saves LlmSettings (from Configuration/LlmSettings.json), which governs model selection, container counts, and plan configurations. - Program registry – maintains ProgramList, a registry of IAiProgramInstance objects that describe the Python programs to be run in containers. Programs are added with AddOrReplaceProgram(). - Container lifecycle – creates, tracks, and destroys AI containers via ExecuteProgramInContainer() and CleanupProgramContainers(). Maintains ContainerList (the list of live containers) and ReservedHostPorts (to prevent port collisions during concurrent creation). - Container pool – owns a ContainerPool instance, started in Start(), stopped in StopContainerPool(), and restarted in StartContainerPool(). - Utilities startup – calls StartUtilities() to bring up the Docker network, Neo4j, the utilities container, and optionally Elasticsearch/Kibana. - URL resolution – GetAiContainerUrl() computes the correct base URL for any container depending on whether the platform is running inside Docker or on the host. - Port management – AllocateFreeHostPort() / ReleaseHostPort() manage the pool of host-side redirected ports available for container-to-host port mapping. - Concurrency – uses ContainerServiceMultithreadSemaphore to serialize concurrent ExecuteProgramInContainer calls, and CleanupSemaphore to serialize concurrent cleanup operations. - Data-changed notifications – exposes OnDataChanged action, invoked whenever the container list changes, to propagate UI refresh events.

Implements: IContainerService, IDisposable

---

ContainerPool (ThalabusCore/Services/ContainerPool.cs)

A background worker service that maintains a pool of warm, Flask-ready AI containers so that session start-up only needs the fast final steps (session-ID handshake) instead of the full container creation sequence.

Key responsibilities: - Background worker – runs WorkerLoopAsync in a background Task, filling the pool up to MaxPreloadedContainers (from LlmSettings.Containers.PreloadedContainers or the constant default) and trimming excess containers. The worker ticks every 30 seconds. - Pool management – maintains an in-memory list of AiContainer objects. Containers are created via CreateOneAsync() → AiContainer.PreloadAsync(), which runs the full warm-up sequence: Docker container creation, code download, Flask exec, Flask readiness poll (GET /ping, up to 50 × 2 s = 100 s), and skill module pre-import (POST /preload/imports). CreateOneAsync adds the container to _pool only after PreloadAsync returns, guaranteeing it is fully warm before any session can acquire it. If PreloadAsync throws (e.g. Flask never responds within 100 s), the container is destroyed and the pool retries on the next tick. Containers are destroyed via DestroyContainerAsync(). - Canonical program – SetCanonicalProgram(program) seeds the pool with a fully-populated IAiProgramInstance (all source files + a main entry-point fragment), then immediately calls NotifySettingsChanged(skipInitialDelay: true) to wake the background worker without re-applying the 60-second startup delay. CreateOneAsync() returns early (no-op) if _canonicalProgram is null, preventing containers from being created before the canonical program is registered. The canonical instance is replaced by the real session program in AssignToProgram() and is never mutated. - Acquire – TryAcquire() returns the first pool container with IsContainerReady == true. Returns null when the pool has no ready container, triggering the slow path in ContainerService. - Assign – AssignToProgram(container, program, isJobContainer) hands a pooled container off to a real program: renames the Docker container, sets AiContainer.AiProgram, sets ContainerInfo.IsJobContainer from the caller, and marks IsFromPool = true. The session ID handshake is performed inside AiContainer.Run() once Flask is ready. - Settings reactivity – NotifySettingsChanged(skipInitialDelay: false) cancels the current sleep so the worker re-evaluates pool size immediately after a settings save. SetCanonicalProgram calls it with skipInitialDelay: true to bypass the 60-second startup delay and fill the pool as soon as the canonical program is registered. - Drain – DrainAsync() destroys all pooled containers in parallel; called on StopAsync() and ReloadAsync(). - Pool container naming – pool containers are named with the prefix defined in Constants.PreloadContainerNamePrefix. IsPoolContainerName() identifies them so CleanupProgramContainers can exclude them from general cleanup.

Used by: ContainerService (exclusively)

---

Container Data Classes

AiContainer (ThalabusCore/Data/Container/AiContainer.cs)

The runtime representation of a running or warming Python AI Docker container. Each instance wraps a Docker container and its multiplexed I/O streams.

Key responsibilities: - Container creation – StartContainerWithInfiniteLoop() creates and starts the Docker container running an infinite bash loop, prepares Docker images and volumes, resolves environment variables, and applies GPU device requests if available. - Pre-loading (Phase A warm-up) – PreloadAsync() performs a complete warm-up: creates the Docker container (infinite bash loop), downloads all source files into it, exec-starts the Flask Python process (always using the main fragment's entry point), attaches stdout/stderr via MultiplexedStream, starts output management and log-tailing background tasks (the output task is stored as OutputManagementTask), polls GET /ping until Flask responds (up to 50 attempts × 2 s = 100 s max), and then POSTs POST /preload/imports to pre-warm the Python sys.modules cache for the key skillsets (ChatBot2, DocIngester, RagIngestion, CrawlerConfluence). On a successful ping, IsContainerReady is set to true. If Flask never responds within 100 s, PreloadAsync throws an AiException, causing CreateOneAsync to destroy the container and retry on the next tick. PreloadAsync enforces an invariant: if AiProgram has no fragments, it throws immediately (programming error). PreloadAsync only returns (normally) once all warm-up steps complete; ContainerPool.CreateOneAsync adds the container to _pool immediately after, guaranteeing it is fully warm before any session can acquire it. - Program execution – Run() has two paths: the standard path (IsFromPool = false) downloads code, exec-starts Flask, attaches streams, and starts background tasks; the pooled fast path (IsFromPool = true) skips all setup (Flask is already running from Phase A), logs reuse, and returns the OutputManagementTask stored during PreloadAsync. Both paths then run the POST /session/{sessionId} retry loop to hand the running Flask its real session ID. - Fragment invocation – InvokeProgramFragment() calls the running Flask server via POST /{skillset}/{fragment}/start/{sessionId} with retry logic (10 attempts × 10-second delays). - Session assignment – SetSessionId() posts POST /session/{sessionId} to update the running Python process with the real session ID after pool reuse. - Stop/cleanup – SendStopCommand() posts POST /main/stop to the Flask server; container teardown is handled by ContainerService.CleanupProgramContainers(). - Output management – streams stdout/stderr from the MultiplexedStream and also tails thalabus.log inside the container. Both tasks read this.ContainerInfo.IsJobContainer dynamically (not captured at task start), so notification behaviour adjusts correctly after AssignToProgram() sets the flag. - Concurrency – uses DockerContainerMultithreadingSemaphore to serialize operations on the same container instance. - Pool markers – IsFromPool is set by ContainerPool.AssignToProgram() to indicate that the container was acquired from the pool rather than freshly created. IsContainerReady is set to true by PreloadAsync() once the Flask /ping succeeds; TryAcquire() filters on this flag as a defense-in-depth guard so only fully warmed containers are handed to sessions.

Implements: IAiContainer, IDisposable

---

LocalAiContainer (ThalabusCore/Data/Container/LocalAiContainer.cs)

A lightweight alternative implementation of IAiContainer for development / local debug scenarios, where the Python server is already running as a local process rather than inside Docker.

Key responsibilities: - Accepts a baseUrl pointing to a locally-running Flask server. - InvokeProgramFragment() delegates directly to the local HTTP endpoint using the same URL pattern as AiContainer. - Run(), SendStopCommand(), and SetSessionId() are no-ops (the process lifecycle is managed externally). - IsFromPool always returns false.

Implements: IAiContainer

---

ContainerUtilities (ThalabusCore/Data/Container/ContainerUtilities.cs)

Manages the utilities container — a persistent, always-on Python container that provides shared services to all sessions.

Key responsibilities: - Network management – CreateContainerNetwork() creates (or re-uses) the Docker bridge network that all containers are attached to. - Utilities startup – StartPythonUtilitiesInContainerAsync() assembles the Python utilities program (all source files returned by Utilities.GetAllSourceFiles()), registers it with ContainerService, and launches it via ExecuteProgramInContainer. Saves the utilities container's IP address as an environment parameter. - Readiness check – StartUtilitiesAsync() polls AiProgram.Ping() until the utilities container responds, throwing if it never becomes ready. - Neo4j launch – StartNeo4j() conditionally creates and starts the Neo4jContainer if the configured NEO4J_URI targets localhost; otherwise skips (remote Neo4j assumed). - Elasticsearch/Kibana launch – StartElasticsearch() / StartKibana() create and start the respective containers. - Settings sync – UpdateLlmSettings() fetches plan configuration from the running utilities container (GET /planconfig) and updates LlmSettings. - Maintenance – TriggerMaintenance() posts POST /maintenance/trigger to the utilities container. - Base URL – GetBaseUrl() returns the correct base HTTP URL for the utilities container, depending on whether running inside Docker or on the host.

Implements: IContainerUtilities, IDisposable

---

Neo4jContainer (ThalabusCore/Data/Container/Neo4jContainer.cs)

Manages the lifecycle of the Neo4j graph database Docker container.

Key responsibilities: - Start() – checks whether a container named Constants.Neo4jContainerName already exists (running or exited). Restarts it if exited; creates and starts it from scratch if absent. Attaches it to the platform Docker network and binds ports 7474 (HTTP) and 7687 (Bolt). - DetermineIpAddress() – inspects the running container to resolve its IP on the platform network. - TestConnection() – opens a Neo4j Bolt session to verify the database is accepting queries. - CleanupNeo4jCache() – runs cache-clearing logic on the database. - Stop() – stops and removes the container.

Implements: INeo4jContainer, IDisposable

---

ElasticsearchContainer (ThalabusCore/Data/Container/ElasticsearchContainer.cs)

Manages the lifecycle of the Elasticsearch Docker container.

Key responsibilities: - Start() – creates and starts the Elasticsearch container on the platform Docker network, binding port 9200. Skips creation if already running. - Mounts a named Docker volume for persistent data storage. - Applies single-node configuration and security settings via environment variables.

Implements: IElasticsearchContainer, IDisposable

---

KibanaContainer (ThalabusCore/Data/Container/KibanaContainer.cs)

Manages the lifecycle of the Kibana Docker container.

Key responsibilities: - Start() – creates and starts the Kibana container on the platform Docker network, binding port 5601. Skips creation if already running. - Configures Elasticsearch connection and credentials via environment variables.

Implements: IKibanaContainer, IDisposable

---

Infrastructure Interfaces (ThalabusInterfaces)

All interfaces reside under ThalabusInterfaces/.

Interface	Location	Purpose
IContainerService	Services/IContainerService.cs	Contract for the central container orchestration service
IAiContainer	Data/Container/IAiContainer.cs	Contract for a running/warming AI container instance
IContainerUtilities	Data/Container/IContainerUtilities.cs	Contract for the utilities container manager
INeo4jContainer	Data/Container/INeo4jContainer.cs	Contract for the Neo4j container manager
IElasticsearchContainer	Data/Container/IElasticsearchContainer.cs	Contract for the Elasticsearch container manager
IKibanaContainer	Data/Container/IKibanaContainer.cs	Contract for the Kibana container manager
ITraceableContainer	Data/Container/ITraceableContainer.cs	Contract for containers that expose chat lines and log files to the UI

---

Supporting Data Structures

ContainerInfo (ThalabusInterfaces/Data/Container/ContainerInfo.cs)

Value object that carries the identity and port mapping of a Docker container: - Id – Docker container ID (SHA-256 prefix). - Name – Docker container name. - IPAddress – IP address on the platform Docker network. - InternalPort – port the Flask server listens on inside the container. - HostRedirectedPort – host port mapped to InternalPort (used when running outside Docker). - IsJobContainer – distinguishes background job containers from interactive session containers. Has public set so ContainerPool.AssignToProgram() can apply the correct value when assigning a pool container to a job or session.

ContainerMessage / IContainerMessage (ThalabusCore/Data/Container/ContainerMessage.cs, ThalabusInterfaces/Data/Container/IContainerMessage.cs)

Rich message objects used to pass structured log entries, skill outputs, and platform notifications from containers to the UI. Carry metadata such as category, message type, severity, expiry, Python source location, goal ID, and Blazor component destination.

AiContainerStatus (enum, in ThalabusInterfaces/Data/Container/IAiContainer.cs)

Lifecycle states for the container subsystem: Starting, Running, Stopping, Stopped, Error.

---

3. Container Activation Sequence (Standard Path)

This is the slow path taken when no pre-loaded container is available from the pool, or when the container is a job container or the utilities container.

sequenceDiagram
    participant Caller
    participant CS as ContainerService
    participant CP as ContainerPool
    participant AC as AiContainer
    participant DE as Docker Engine

    Caller->>CS: ExecuteProgramInContainer(programId)
    activate CS
    Note over CS: Acquire ContainerServiceMultithreadSemaphore
    CS->>CS: CleanupProgramContainers()
    Note over CS: Pool is skipped for job containers and the utilities container
    CS->>CP: TryAcquire()
    CP-->>CS: null — pool empty
    CS->>CS: GetFreeHostPort()
    CS->>AC: new AiContainer(program, hostPort, internalPort)
    CS->>CS: ContainerList.Add(aiContainer)

    CS->>AC: Run(targetFragmentName, destroyContainer)
    activate AC
    Note over AC: Acquire DockerContainerMultithreadingSemaphore

    Note over AC: IsFromPool = false → StartContainerWithInfiniteLoop()
    AC->>AC: AiProgram.Prepare(targetFragment)
    AC->>DE: PrepareImages() — pull / tag image
    AC->>DE: PrepareVolumes() — ensure named volumes
    AC->>DE: CreateContainerAsync(image, cmd=[bash infinite loop], env, portBindings, volumes)
    DE-->>AC: containerId
    AC->>DE: StartContainerAsync(containerId)
    AC->>DE: InspectContainerAsync() — DetermineIpAddress
    DE-->>AC: IPAddress

    AC->>DE: DownloadProgramIntoContainer() — tar upload (ExtractArchiveToContainerAsync) + chown + pycache cleanup
    AC->>DE: ExecContainerProgram() — docker exec python (always main fragment entry point)
    AC->>DE: StartAndAttachContainerExecAsync() → MultiplexedStream
    AC-->>AC: StartContainerOutputManagement() [background Task]
    AC-->>AC: StartContainerLogFileTailing() [background Task]
    Note over AC: Release DockerContainerMultithreadingSemaphore
    deactivate AC

    Note over CS: Release ContainerServiceMultithreadSemaphore
    CS->>CS: OnDataChanged.Invoke()
    CS-->>Caller: IAiContainer
    deactivate CS

Key invariant: The container always starts running an infinite bash sleep loop first. The Python process is injected later via docker exec, so the same startup code path is shared by both the standard path and the pre-loaded pool path.

---

4. Pre-Loaded Container Activation Sequence

The ContainerPool eliminates cold-start latency by keeping warm containers ready. The sequence has two distinct phases: pool fill (background, proactive) and pool acquisition (on demand, fast).

Phase A — Pool Fill (Background Worker)

sequenceDiagram
    participant CP as ContainerPool
    participant CS as ContainerService
    participant AC as AiContainer
    participant DE as Docker Engine
    participant Flask

    Note over CP: WorkerLoopAsync starts
    CP->>CP: Wait ContainerPoolStartupDelaySeconds (skipped on ReloadAsync and SetCanonicalProgram)

    loop Every 30 seconds (or on wake from SetCanonicalProgram or NotifySettingsChanged)
        CP->>CS: LlmSettings.Containers.PreloadedContainers
        CS-->>CP: maxPreloadedContainers

        alt pool.Count < max AND _canonicalProgram != null
            CP->>CS: AllocateFreeHostPort()
            CS-->>CP: hostPort
            CP->>AC: new AiContainer(canonicalProgram, hostPort, internalPort)
            CP->>AC: PreloadAsync(preloadName)
            activate AC
            Note over AC: Acquire DockerContainerMultithreadingSemaphore
            AC->>DE: PrepareImages() — pull / tag image
            AC->>DE: PrepareVolumes() — ensure named volumes
            AC->>DE: CreateContainerAsync(name=preloadName, cmd=[bash infinite loop], SESSION_ID=placeholder)
            DE-->>AC: containerId
            AC->>DE: StartContainerAsync(containerId)
            AC->>DE: InspectContainerAsync() — DetermineIpAddress
            DE-->>AC: IPAddress
            AC->>DE: DownloadProgramIntoContainer() — tar upload + chown + pycache cleanup
            AC->>DE: ExecContainerProgram() — docker exec python (main entry point)
            AC->>DE: StartAndAttachContainerExecAsync() → MultiplexedStream
            AC-->>AC: StartContainerOutputManagement() [background Task — stored as OutputManagementTask]
            AC-->>AC: StartContainerLogFileTailing() [background Task]
            loop Poll until ready (up to 50 × 2 s = 100 s max)
                AC->>Flask: GET /ping
                Flask-->>AC: 200 OK
            end
            alt Flask responded within 100 s
                AC->>AC: IsContainerReady = true
                AC->>Flask: POST /preload/imports (pre-warm sys.modules for ChatBot2, DocIngester, RagIngestion, CrawlerConfluence)
                Flask-->>AC: 200 OK (all imports complete — synchronous, blocks until done)
                Note over AC: Release DockerContainerMultithreadingSemaphore
                deactivate AC
                CP->>CP: _pool.Add(container)
                Note over CP: Container is warm — Flask running, code loaded, skill modules pre-imported
            else Flask timeout (100 s exceeded)
                AC-->>CP: throw AiException
                CP->>DE: DestroyContainerAsync() — stop + remove container, release port
                Note over CP: Retry on next worker tick
            end
        else _canonicalProgram == null
            Note over CP: Skip — canonical program not registered yet
            Note over CP: SetCanonicalProgram() will wake the worker with skipInitialDelay=true
        end
    end

Phase B — Pool Acquisition (Session Start, Fast Path)

sequenceDiagram
    participant Caller
    participant CS as ContainerService
    participant CP as ContainerPool
    participant AC as AiContainer (pooled)
    participant DE as Docker Engine
    participant Flask

    Caller->>CS: ExecuteProgramInContainer(programId)
    activate CS
    Note over CS: Acquire ContainerServiceMultithreadSemaphore
    CS->>CS: CleanupProgramContainers()
    CS->>CP: TryAcquire()
    CP-->>CS: AiContainer (IsContainerReady=true — Flask already running)
    CP->>CP: OnDataChanged.Invoke() — UI refresh

    CS->>CP: AssignToProgram(container, program, isJobContainer)
    CP->>DE: RenameContainerAsync(newName = program.GetContainerName())
    CP->>AC: AiProgram = program, IsFromPool = true
    CP->>AC: ContainerInfo.IsJobContainer = isJobContainer
    CS->>CS: ContainerList.Add(container)
    Note over CS: Release ContainerServiceMultithreadSemaphore

    CS->>AC: Run(targetFragmentName, destroyContainer)
    activate AC
    Note over AC: IsFromPool = true — Flask already running, skip all setup
    AC->>AC: Log - Reusing pre-loaded container...
    loop up to 20 attempts × 2 s
        AC->>Flask: POST /session/{sessionId}
        alt 200 OK
            Flask-->>AC: session confirmed
        else not ready yet
            Flask-->>AC: error / timeout — retry
        end
    end
    deactivate AC

    CS->>CS: OnDataChanged.Invoke()
    CS-->>Caller: IAiContainer
    deactivate CS

Key difference from the standard path: Flask is already running in the pool container (started during Phase A), guaranteed by IsContainerReady == true before TryAcquire returns. Phase B only renames the container, sets the job/session flag, and sends the real session ID. No Docker container creation, no code download, no Python exec, and no stream attachment occurs at session start time. The 20-attempt retry loop for POST /session/{sessionId} is a defensive measure; the first attempt succeeds in the normal case since IsContainerReady guarantees Flask is up.

---

5. Startup Orchestration (Program.cs)

ThalabusWeb/Program.cs coordinates the overall boot sequence:

sequenceDiagram
    participant P as Program.cs
    participant CS as ContainerService
    participant CU as ContainerUtilities
    participant CP as ContainerPool
    participant DE as Docker Engine
    participant Neo4j
    participant SS as SkillsService

    P->>CS: AddSingleton (DI registration)
    Note over CS: constructor — read env vars, load EnvironmentVariables.json, load LlmSettings.json

    P->>CS: Init()
    Note over CS: Mount Docker volumes (Windows host), create ContainerPool (not started yet)

    P->>SS: LoadSkillsets()

    P->>CS: Start()
    activate CS

    CS->>DE: DockerPing() — verify Docker Engine is reachable
    CS->>CS: Cleanup() + CleanupProgramContainers() — remove stale containers

    CS->>CU: CreateContainerNetwork() — create / reuse bridge network
    CS->>CU: StartNeo4j()
    activate CU
    CU->>DE: CreateContainerAsync / StartContainerAsync (Neo4j)
    DE-->>CU: running
    CU->>Neo4j: TestConnection() — poll until ready
    Neo4j-->>CU: OK
    deactivate CU

    CS->>CU: StartUtilitiesAsync() — start Python utilities container
    activate CU
    CU->>DE: CreateContainerAsync / StartContainerAsync (utilities)
    DE-->>CU: running
    CU->>CU: Ping() — poll until Flask ready
    deactivate CU

    opt ELASTICSEARCH_PASSWORD is set
        CS->>CU: StartElasticsearch()
        CU->>DE: CreateContainerAsync / StartContainerAsync (Elasticsearch)
        CS->>CU: StartKibana()
        CU->>DE: CreateContainerAsync / StartContainerAsync (Kibana)
    end

    CS->>Neo4j: DeleteAllSessionsAndDatasets()
    CS->>Neo4j: CreateDefaults(LlmSettings.KbDefaults)
    CS->>CS: CleanupNeo4jCache()

    CS->>CP: StartAsync()
    Note over CP: WorkerLoopAsync starts in background — begins filling pre-loaded pool

    deactivate CS

After Start() returns, the platform is fully operational: infrastructure containers are running, the utilities container is ready, the Neo4j schema defaults exist, and the pool worker is filling warm containers in the background.

SetCanonicalProgram registration: when AiProgramInstance.Execute() calls ContainerService.AddOrReplaceProgram() with a non-utility program that has source fragments, ContainerService automatically calls ContainerPool.SetCanonicalProgram(). This both seeds the pool worker with a canonical program instance for Phase A warm-up AND immediately wakes the worker via NotifySettingsChanged(skipInitialDelay: true), bypassing the 60-second startup delay so the pool starts filling right away. Before a canonical program is registered, CreateOneAsync returns without creating any container.