Technical Architecture & Engineering Blueprint

A declarative YAML-to-CSX integration testing platform for distributed .NET systems (engine on .NET 8 LTS)

Technical Architecture

Document	Architecture & Engineering Blueprint
Platform	E2E Microservices Integration Testing SaaS
Version	1.0 (Draft for Review)
Status	For engineering review
Audience	Architects, platform engineers, technical leadership
Date	May 2026

---

[Table of contents auto-generated by viewer; see section list at top of repo.]

1. Purpose and Scope of this Document

This blueprint defines the engineering architecture of the End-to-End (E2E) Microservices Integration Testing SaaS platform. Where the originating market-and-architecture analysis established why the platform should exist and who it serves, this document concentrates on how it is built. It is intended to be precise enough that an engineering team can use it as the authoritative reference when decomposing the system into services, selecting libraries, and writing acceptance criteria.

The platform addresses a structural gap in the .NET ecosystem. Unit tests verify isolated components but say nothing about the network of dependencies that defines a modern distributed system, while traditional end-to-end frameworks are fragile, slow, and expensive to maintain. When a single business transaction travels through a REST call, a Kafka event, a database mutation, and an outbound webhook, conventional approaches collapse under the weight of orchestrating the infrastructure. The architecture described here treats that orchestration as a first-class, automated concern rather than as incidental test setup.

Design thesis

Declarative test intent (authored in YAML) is compiled into Turing-complete execution power (C# scripts run through Roslyn), and that execution is wrapped in fully automated infrastructure orchestration (.NET Aspire and Testcontainers). The three layers are independent enough to evolve separately, yet contractually bound through a single typed object model.

1.1 How to read this document

Sections 2 and 3 give the system context and the layered architecture overview. Sections 4 through 6 describe the three core engines: orchestration, the Roslyn compiler, and the cloud execution fabric. Section 7 covers the three deployment topologies the same test suite must run against unchanged. Sections 8 through 10 cover agentic automation, the performance-testing capability, and resilience. Sections 11 and 12 address security and observability as cross-cutting concerns. Section 13 defines the provider plugin architecture through which the DSL extends to new technologies by community contribution at the source level. Section 14 defines the result-reporting surface through which the platform's outcome reaches developers, CI systems, team leads, and auditors. Sections 15 through 17 cover three operational concerns that determine whether the platform is usable in practice: test data and environment seeding, the test runner and its selection model, and secrets and configuration management. Section 18 records the principal engineering risks. Readers who want the companion specification of the YAML language itself and the VSCode tooling should consult the YAML DSL Specification & VSCode Extension Design document; that material is deliberately not duplicated here.

1.2 Validation log

This blueprint has been progressively validated against working code through a sequence of small, time-boxed spikes. The table records what each spike retired or corrected so that a reader can see which parts of the architecture are now grounded in working evidence rather than design intent alone. Each spike's corrections are folded into the relevant sections below; this table is the index to those corrections.

Spike	What it covered	What it retired	What it corrected
Spike 1	End-to-end vertical slice: one YAML → one container → one Roslyn-compiled step → one verdict, on .NET 8 LTS / Aspire 13.x.	The risks that Aspire's programmatic builder would not behave as the architecture assumed, and that Roslyn scripting would lag the .NET 8 LTS language version.	The illustrative AppHost code in §4.1 (headless mode requires DisableDashboard, and endpoints resolve through the IResourceBuilder returned by AddContainer/AddProject, not through app.GetEndpoint(name, scheme)).
Spike 2	Provider contract and fragment composition: two providers (http.rest and db-assert.postgres) composed into one Roslyn script, on .NET 8 LTS / Aspire 13.3.5 / Npgsql 10.0.2.	The risks that the provider contract would not survive contact with real code, that fragment composition would produce string-splicing landmines, and that the database-resource lifecycle could not be relied upon.	The CsxFragment composition contract (§13.3.1) now records the no-using-var constraint and the C# 11 $$"""…""" raw-string emit-template idiom; the worked Postgres provider (§13.10) shows the IResourceWithConnectionString cast and the sanitised-identifier pattern; §4.3 distinguishes server-level from database-level health gating.

Subsequent spikes are added to this table as they complete. The pattern is deliberate: the architecture is allowed to be aspirational at first, and is required to converge on what code actually does as evidence accumulates. A document that records its relationship to the code it describes is more trustworthy than a document that does not.

2. System Context and Design Goals

Before describing internal structure, it is worth fixing the boundary of the system: what sits inside it, what it depends on, and what it must never assume. A test platform is unusual in that the software it exercises is supplied entirely by the customer, so the design must remain agnostic to the customer's domain while being opinionated about how infrastructure is provisioned.

2.1 Actors and external systems

Four classes of actor interact with the platform, and each shapes a different part of the architecture.

Actor / system	Relationship to the platform	Architectural consequence
Test author (developer or QA engineer)	Writes YAML test definitions and, where needed, embedded C# script blocks; runs suites locally or against the cloud.	Drives the DSL design, the VSCode tooling, and the requirement that local and remote execution be behaviourally identical.
Customer microservices and infrastructure	The system under test — REST APIs, Kafka topics, SQL Server / PostgreSQL / MongoDB stores, Elasticsearch indices, outbound webhooks.	The platform must provision these as containers and discover their endpoints dynamically; it must make no assumption about the customer's schemas.
CI/CD pipeline	A non-interactive runner (GitHub Actions, Azure DevOps, GitLab CI) that invokes the test runner as part of a build.	Requires a headless CLI, deterministic exit codes, and the ability to offload heavy containers from a runner that lacks nested virtualisation.
Enterprise identity provider	A SAML 2.0 / OIDC IdP that authenticates users and authorises container provisioning in regulated environments.	Authentication is not confined to a web dashboard; it reaches into the CLI and the IDE plugin and gates resource-intensive operations.

2.2 Design goals

The architecture is evaluated against seven goals. They are listed in priority order, because several of them trade against one another and the ordering tells an engineer which way to resolve a conflict.

1. Topological parity. An identical test suite must execute unchanged on a developer laptop, in the SaaS cloud, and inside a CI pipeline. Parity outranks raw performance, because eliminating the “works on my machine” failure mode is the platform's central promise.
2. Determinism under asynchrony. Distributed tests are inherently flaky; the engine, not the test author, owns retry, polling, and backoff so that results do not depend on hardware speed.
3. Bounded resource consumption. Dynamically compiled scripts must be fully reclaimable; a long-running suite must not leak assemblies, memory, or orphaned containers.
4. Isolation. Concurrent test runs, and concurrent tenants, must not observe one another's data, ports, or containers.
5. Extensibility without forking. Customers must be able to inject proprietary libraries and domain models into a test without the platform team shipping a new release.
6. Security by construction. In the enterprise topology, identity, authorisation, and audit are part of the execution path rather than a wrapper around it.
7. Operability. Every layer emits structured telemetry so that a failed test can be distinguished from a failed environment.

2.3 Explicit non-goals

Stating non-goals prevents scope creep and clarifies interfaces. The platform is not a unit-testing framework and does not replace xUnit, NUnit, or MSTest for in-process assertions. It is not a UI or browser automation tool; it validates service-to-service and data-tier behaviour, not rendered front-ends. It does not author the customer's services or generate production code. Finally, it does not host the customer's production infrastructure — every container it starts is ephemeral and exists only for the lifetime of a test run.

3. High-Level Architecture

The platform is best understood as five cooperating layers. A test definition enters at the top as declarative YAML and descends through compilation, orchestration, and execution, with results and telemetry rising back up. Each layer exposes a narrow, typed contract to the layer above it, which is what allows, for example, the cloud execution fabric to be swapped for local execution without the compiler noticing.

3.1 The five layers

Layer	Responsibility	Principal technologies
1. Authoring	Express test intent and surface errors before execution.	YAML DSL, JSON Schema, VSCode extension, Language Server Protocol.
2. Compilation	Translate validated YAML into a cached, executable C# delegate.	YAML parser, AST, CSX code generation, Roslyn Scripting API.
3. Orchestration	Provision the system under test and its dependencies; resolve endpoints.	.NET Aspire AppHost, Testcontainers, health-gated startup.
4. Execution	Run the compiled delegate against the orchestrated topology and collect assertions.	Roslyn delegate host, collectible AssemblyLoadContext, Polly resilience.
5. Infrastructure fabric	Decide where containers physically run — local socket, SaaS cloud, or on-premises cluster.	Docker socket proxy, SSH tunnelling, Ryuk reaper, Kubernetes.

Table 3.1 — The five architectural layers and their contracts.

3.2 The canonical request flow

The clearest way to internalise the architecture is to follow a single test from authoring to verdict. The example below is the platform's reference scenario: a transaction that crosses four technologies.

1. Author. A developer writes an .e2e.yaml file describing a POST to a user-creation API, an expected Kafka event on a topic, an expected MongoDB document, and an expected outbound webhook.
2. Validate. The VSCode extension validates the file against the platform's JSON Schema in real time and cross-references the referenced API path against the project's OpenAPI specification.
3. Compile. On execution, the YAML is parsed into an Abstract Syntax Tree, the AST is translated into a CSX script, and Roslyn compiles that script once into an in-memory delegate.
4. Orchestrate. The Aspire AppHost starts the API container plus Kafka, MongoDB, and a schema registry, waits for each to report healthy, and resolves their connection strings.
5. Execute. The compiled delegate runs inside a collectible load context. It issues the HTTP POST, polls Kafka for the event, queries MongoDB for the document, and listens on an ephemeral local port for the webhook.
6. Verify and reclaim. Assertions are collected into a structured result; the load context is unloaded; Respawn resets the databases; the verdict and telemetry are returned to the runner.

   Why the layering matters

   Because the compiler emits a delegate that depends only on a typed global-context object, and the orchestration layer is the sole producer of that object, the same compiled test is indifferent to whether its Kafka broker runs on the laptop or in a remote VPC. Topological parity — design goal one — is therefore a property of the layer contract, not something each test must re-earn.

4. Orchestration Engine: .NET Aspire and Testcontainers

The orchestration layer replaces the brittle scaffolding that has historically surrounded integration tests — hand-written docker-compose files, shell scripts that sleep and hope, and boilerplate-heavy WebApplicationFactory harnesses. Its job is to turn a declared list of dependencies into a running, healthy, discoverable topology, and then to tear it down cleanly.

4.1 Building the application host programmatically from the YAML

The engine constructs the test environment programmatically rather than hosting a pre-existing AppHost project. At suite startup the runner reads the YAML's environment section and assembles a DistributedApplication by calling Aspire's builder API directly: each entry under services becomes a project or container resource, each entry under dependencies becomes a typed managed resource (Postgres, Kafka, MongoDB, Redis, RabbitMQ, Elasticsearch, and so on). Service discovery and randomised proxy ports are still Aspire's responsibility, which is what permits many test runs to execute concurrently on one machine without port collisions; the test code never extracts an IP address or port by hand.

The customer therefore brings exactly one of two things for each service in their YAML: a container image reference or a path to a csproj. The image path is the simplest and fastest — the engine calls AddContainer with the image reference and Aspire pulls and runs it. The csproj path is for teams iterating on a service locally — the engine calls AddProject with the path and Aspire builds the project as part of suite startup. The strongly-typed AddProject() generic variant, which requires a pre-existing AppHost csproj and source-generator metadata, is deliberately not offered, because it imports a level of compile-time coupling that contradicts the YAML-first premise. The image form is the recommended default.

// Illustrative: the engine constructs an Aspire host from the YAML.
// The headless test runner disables the Aspire dashboard, because
// the dashboard requires ASPNETCORE_URLS and ASPIRE_DASHBOARD_OTLP_*
// environment variables that only the 'aspire run' tooling injects;
// a bare 'dotnet run' otherwise throws in BeforeStart.
var builder = DistributedApplication.CreateBuilder(
    new DistributedApplicationOptions { DisableDashboard = true });

// From environment.dependencies:
var ordersDb = builder.AddPostgres("orders-db")
                      .WithImageTag("16");
var events   = builder.AddKafka("events")
                      .WithKafkaUI();

// From environment.services — image form. The IResourceBuilder
// returned by AddContainer is held so endpoints can be resolved
// from it after the host starts (see below).
var ordersApi = builder.AddContainer("orders-api", "myorg/orders-api:1.2.3")
                       .WithHttpEndpoint(targetPort: 8080)
                       .WithReference(ordersDb).WithReference(events)
                       .WaitFor(ordersDb).WaitFor(events);

// Or, csproj-by-path form for a service still being built locally:
// var ordersApi = builder.AddProject("orders-api",
//                                    "./src/Orders.Api/Orders.Api.csproj")
//                        .WithReference(ordersDb).WithReference(events)
//                        .WaitFor(ordersDb).WaitFor(events);

await using var app = builder.Build();
await app.StartAsync();

// Wait for healthy. For resources with a health check this blocks
// until the check passes; for resources without one it returns
// as soon as the resource reaches Running.
await app.ResourceNotifications
         .WaitForResourceHealthyAsync("orders-api");

// Endpoints and connection strings are resolved, never hard-coded.
// Dependencies expose connection strings on the app:
var ordersDbConn = app.GetConnectionString("orders-db");
// Services expose endpoints on the IResourceBuilder retained above:
var ordersApiUrl = ordersApi.GetEndpoint("http").Url;

Two API distinctions are worth fixing in mind because they appear throughout the generated code. app.GetConnectionString(name) returns connection strings for managed resources (databases, brokers, caches) and is what db-assert.* and mq-* providers consume. For services, the endpoint URL is resolved through the IResourceBuilder returned by AddContainer or AddProject — the runner retains that builder per service and calls builder.GetEndpoint("http").Url (or "https") after StartAsync returns. The earlier form app.GetEndpoint(name, scheme) does not exist in current Aspire releases; this was discovered during an early vertical-slice spike on Aspire 13.x and the runner's code generator uses the resource-builder form throughout. Conflating connection strings and endpoints is the most common error in hand-rolled Aspire setups; the engine avoids it by construction because each provider declares which form it needs.

A note on the dashboard. Aspire's DistributedApplicationBuilder defaults to launching the Aspire dashboard, which requires environment variables that the aspire run tooling normally injects and that dotnet run does not. Because the test runner is a headless process invoked from a CLI or a CI pipeline, the runner constructs the builder with DisableDashboard = true, as shown above. An adopter who instead launches the runner from within an Aspire AppHost project (a less common path the platform also supports) inherits the dashboard from that AppHost; the headless option matters only for the bare invocation.

4.1.1 Image references, registry resolution, and pull policy

Container image references in the YAML follow the standard Docker image reference format, which means any OCI-compliant registry is reachable without Aspire-specific configuration: Docker Hub for un-prefixed names, GitHub Container Registry for ghcr.io-prefixed names, Azure Container Registry for myco.azurecr.io-prefixed names, ECR, GCR, Harbor, JFrog Artifactory, and so on. Authentication is delegated to the underlying container runtime: when a developer runs docker login (or the CI step's equivalent), credentials are stored in the runtime's credential store and used automatically for subsequent pulls. The engine does not reimplement Docker authentication, and it does not need to.

Three controls on top of the image reference are exposed to authors. An imageRegistry override at the environment level prefixes every un-prefixed image with the given registry hostname, which is what teams in regulated environments use to redirect public images to an internal mirror. An imagePullPolicy field accepts the standard Docker values — Always, Missing, Never — with the engine choosing a sensible default per image (Always for moving tags such as :latest or :main, Missing for semver-shaped tags). Explicit digest pinning with @sha256:… is supported and is the recommended form for production-grade reproducibility, because tag-only references can move under the team's feet whereas digests are byte-stable. The reproducibility envelope of Section 14.7 records the resolved digest for every container the suite uses regardless of whether the YAML pinned it, so any run can be reproduced exactly.

Image-pull failures are environment errors, not test failures

When the runtime cannot pull an image — registry unreachable, authentication failed, image not found — the orchestrator classifies the resulting failure as an environment error per the verdict taxonomy in Section 12.1. The report names the registry hostname and the authentication status alongside the underlying runtime error, so an engineer can act on it without spelunking. The authentication status distinguishes five cases: "unauthenticated" (401 / rejected credentials), "access-denied" (403 / forbidden), "rate-limited" (429 / Docker Hub cold-runner failure), "anonymous" (pull without credentials, non-auth failure), and null (registry unreachable via DNS or network, no authentication attempted). This granularity — especially the distinction between "rate-limited" and "anonymous" — lets engineers fix the right failure without trial and error. Image-pull failures are one of the highest-frequency causes of pilot frustration in any Docker-based testing tool, and naming them explicitly is cheap insurance.

4.2 Performance: build the topology once per scenario or once per suite

Spinning up SQL Server or a Kafka broker for every test scenario is the single largest source of slow pipelines. The engine amortises this cost through a TopologyFixture managed by the runner: a fixture is constructed when the runner first needs the topology a scenario declares, is shared across scenarios that declare the same topology and have not opted out of sharing, and is torn down only when no remaining selected scenario needs it. The fixture's lifecycle hooks — build, start, mark healthy, reset state, dispose — are owned by the runner rather than by an xUnit collection fixture, because the runner is bespoke (see Section 16) and its scheduling decisions are not legible to xUnit's test-host model. The contract is the same one xUnit's fixtures provide; the implementation is the platform's own.

The default in sequential mode is one topology per suite, with Respawn-based state reset between scenarios providing the isolation. In parallel mode the rules change because parallel scenarios cannot safely share a mutable topology; the runner's isolation contract in Section 16.3 governs the parallel case. Either way, the heavy infrastructure is paid for at most once per scenario and typically once per suite, and the fixture is reused across every iteration of every test the suite contains.

4.2.1 Watch mode for local iteration

For developers iterating on a single .e2e.yaml file, the CLI runner's --watch flag compresses the author→run loop by eliminating repeated topology rebuilds. When invoked with vouchfx run <file> --watch, the runner: builds the topology once, runs the scenario, then watches the file for changes. On each save, it re-compiles the scenario and decides whether to re-use the kept topology or rebuild it, based on whether the environment block hash changed. If the hash is unchanged (a steps-only edit, the common case during iteration), the scenario re-runs against the existing topology; if the hash changed, the old topology is disposed, a new one is built, and the scenario runs against it. The reuse path first resets the database via Respawn, then re-applies the seed block to restore the freshly-provisioned baseline — matching the baseline of a plain vouchfx run so each watch run starts from identical state. Compile errors and run failures are reported concisely and the watch loop keeps running, so the developer can edit again without re-running from the shell. The --watch and --parallel flags are mutually exclusive (combining them is a usage error), as watch mode targets the single-file edit-run loop whilst parallelism fans multiple scenarios across independent topologies.

4.3 Startup determinism

Microservices frequently crash on boot if their backing store is not yet accepting connections. A naive harness races against this and produces intermittent, irreproducible failures. The engine eliminates the race by modelling the dependency graph explicitly as a Directed Acyclic Graph and enforcing startup order with Aspire's .WaitFor(...) construct. The test runner then blocks on app.ResourceNotifications.WaitForResourceHealthyAsync("resource-name") with a bounded cancellation timeout, so that no test step fires until the entire distributed graph reports healthy. A timeout here is reported as an environment failure, not a test failure — a distinction that section 12 treats as essential.

A subtlety worth recording: wait on the resource the test depends on, not the resource above it. Aspire models a Postgres dependency as two distinct resources: a server resource (the postgres container itself, healthy as soon as it accepts connections) and a database resource (a specific database created by Aspire's lifecycle script, healthy only once the database exists). A test that queries a named database must wait on the database resource, not the server. Waiting on the server returns as soon as Postgres accepts connections, which is before Aspire has run the script that creates the database; a subsequent query then fails intermittently — depending on whether script compilation and startup happen to be slow enough to let the database appear before the first query lands. This race was discovered during an early provider-contract spike: the run passed only because Roslyn compilation was slow enough to let the database get created during the gap, and the same code would have failed intermittently on faster hardware. The runner's code generator therefore always waits on the most specific resource a step depends on — the database, the topic, the named queue — rather than the server above it. The same principle applies to Kafka topics, RabbitMQ queues, and any other case where a managed resource has a finer-grained lifecycle than its host.

Health-check log noise is filtered at the runner level. Aspire's resource notifications wait for genuine health, but the health-check pollers underneath emit per-attempt failure logs at Error level while a resource is still booting (“Postgres starting up”, “database does not exist”). These are cosmetic transient startup messages, not real failures, but they make a passing run look broken. The runner therefore configures the host's logging to suppress Microsoft.Extensions.Diagnostics.HealthChecks output below Warning by default, and exposes a flag for adopters who want the verbose form for debugging.

4.4 Container lifetime and state reset

Two policies govern how containers live and how their data is kept clean between tests, and they are deliberately separated because they answer different questions.

4.4.1 Persistent container lifetime

For suites that run repeatedly on the same machine — a developer iterating locally, or a long-lived CI agent — the engine supports the ContainerLifetime.Persistent policy. A heavy database container can then retain its filesystem across distinct test executions, so subsequent runs skip the multi-second bootstrap of the image. This is a latency optimisation and is never the mechanism for test isolation.

4.4.2 Deterministic state reset

Isolation between tests is achieved by per-store cleanup invoked during the runner's between-scenario hook, never by recreating the container. The mechanism varies by store technology and is owned by the corresponding provider:

Store type	Reset mechanism
PostgreSQL, SQL Server, MySQL	Respawn: inspects relational metadata, computes a deletion order respecting foreign-key constraints, and issues DELETE statements. Tables and schemas are preserved; identity and sequence values are not reset. Returns to baseline in milliseconds.
MongoDB	Delete per-collection via deleteMany with no filter; system collections and views are skipped; collection indexes are preserved. Capped and time-series collections fail the reset (environment error).
Redis	FLUSHDB against the database designated by the discovered connection string; other databases on the same instance are untouched.
Elasticsearch	Delete via _delete_by_query matching all documents across open indices with conflicts=proceed&refresh=true&expand_wildcards=open. Mappings and settings are preserved; hidden and system indices are excluded. Per-document failures or timeout fail the reset.
Kafka, RabbitMQ, NATS, Azure Service Bus	Not applicable — messages are consumed per step; scope topics/queues/subjects per suite.
DynamoDB, MinIO	Not reset — add explicit cleanup steps.

All of these resets are invoked through the same runner hook and produce the same verdict-taxonomy classification on failure (a reset failure is an environment error, never a test failure). The architectural point is that container lifetime and state lifetime are separately managed, and conflating them — for example, recreating containers to get a clean database — destroys the latency optimisation and is explicitly disallowed.

4.5 Teardown discipline

The headless topology host (the single chokepoint through which all suite and stub topologies are disposed) must follow a strict teardown sequence to ensure DCP deletes all containers and networks before the process exits. A naive DisposeAsync() call alone does not stop the application; in Aspire 13.4.2, it leaves the DCP lifecycle manager running in the background. The default DcpPublisher:WaitForResourceCleanup setting is false, meaning DCP's stop signal fires asynchronously and returns immediately, racing the process exit. The detached DCP apiserver then finishes deletion after the process has gone, and the containers and aspire-session-network-* network are orphaned on the host, accumulating across test runs.

The invariant is therefore: (1) set builder.Configuration["DcpPublisher:WaitForResourceCleanup"] = "true" during StartAsync, before builder.Build(), so DCP will synchronously delete resources; and (2) in DisposeAsync, call a bounded await app.StopAsync(...) (with a 15-second cancellation timeout) before the existing await app.DisposeAsync() call. This mirrors the pattern used in Aspire's own Aspire.Hosting.Testing.DistributedApplicationFactory, which is the only teardown path in the Aspire box that does not leak containers. The bounded timeout sits above DCP's own internal cleanup timeout (approximately 10 seconds), so normal operation completes well before the deadline; the timeout acts as a circuit-breaker against a wedged stop. Exceptions thrown during StopAsync are swallowed — teardown must never throw into the verdict path (§12.1) — and the subsequent DisposeAsync is still invoked as the final resource release.

5. The Roslyn CSX Compiler and Memory Management

The compilation and execution layer is where the platform's central tension is resolved. Test authors want the brevity of declarative YAML; edge cases demand the full power of a real programming language. The engine satisfies both by generating C# script from YAML and executing it through Roslyn. The difficulty is that executing dynamically generated code inside a long-running process is a notorious source of unbounded memory growth, and the architecture must treat that as a primary concern rather than an afterthought.

5.1 The compilation pipeline

Compilation proceeds through five stages. The DSL specification document defines the YAML grammar in full; here the focus is on the mechanical translation and on how each stage protects a design goal.

Stage	What happens	Goal protected
Lexical analysis and parsing	YAML is parsed into an Abstract Syntax Tree and validated against the platform JSON Schema; structurally invalid suites fail before any container starts.	Determinism; fast feedback.
CSX code generation	AST nodes become optimised C#: HTTP steps become HttpClient blocks bound to Aspire discovery; Kafka steps become Confluent producer/consumer delegates with Avro and schema-registry support.	Topological parity.
Roslyn compilation	The generated CSX string is compiled in-memory once via the Roslyn Scripting API into a reusable, high-performance delegate.	Bounded resource consumption.
Global context injection	A strongly-typed host object carries pre-built HTTP clients, Kafka consumers, connection strings, and captured variables into the script's scope.	Isolation; parity.
Dynamic assembly resolution	Customer NuGet packages and domain DLLs are added to the compilation so scripts can use proprietary object models.	Extensibility without forking.

5.2 The memory-leak failure mode

The obvious way to run a script — calling CSharpScript.EvaluateAsync() — is also a trap. Every invocation emits a fresh, anonymous dynamic assembly into the process. Because the classic .NET runtime cannot unload an individual assembly, a suite that runs thousands of iterations accumulates uncollectable assemblies until it terminates with an OutOfMemoryException. For a platform whose explicit purpose includes stress and load testing, this is disqualifying. The architecture therefore mandates a different pattern.

5.3 The mandated pattern: compile once, isolate, unload

Three techniques combine to make dynamic execution fully reclaimable. They must be used together; any one alone is insufficient.

5.3.1 Pre-compilation to a byte stream

Instead of evaluating the script, the compiler calls CSharpScript.Create<T>() and then .Emit() to write the generated assembly into a MemoryStream. Compilation happens exactly once; the result is a delegate obtained through .CreateDelegate() that can be invoked repeatedly with no recompilation overhead. This is what makes high-iteration load scenarios viable.

5.3.2 Collectible AssemblyLoadContext

The emitted byte array is loaded into a custom class that inherits from AssemblyLoadContext and is constructed with isCollectible: true. A collectible load context can be unloaded as a unit, taking every assembly it owns with it. When a test scenario finishes, the engine calls .Unload() on the context and disposes the backing stream; the garbage collector then reclaims all dynamically generated code. Memory therefore returns to baseline after each scenario regardless of how many iterations it ran.

5.3.3 Global context as the only bridge

The compiled script reaches the orchestrated environment exclusively through a strongly-typed ScriptGlobalVariables host object. It carries pre-configured HTTP clients, active Kafka consumers, database connection strings, a mutable dictionary of variables captured from earlier steps, a secrets accessor, and a webhook-capture accessor. Because this object is the single, narrow channel between the durable engine and the disposable script, unloading the load context severs every reference cleanly — there are no stray static handles to keep an assembly alive. The script reads captured inbound webhook requests (for webhook-listen.http steps) through ScriptGlobalVariables.Webhooks (IWebhookCaptureAccessor.GetCaptured(listenerName) → captured requests), an instance member exactly like Secrets, so the collectible AssemblyLoadContext boundary remains clean: no static handle crosses it.

// The compile-once / isolate / unload contract (illustrative)
var script   = CSharpScript.Create<TestResult>(csxSource, scriptOptions,
                                               typeof(ScriptGlobalVariables));
var compiled = script.GetCompilation();

using var ms = new MemoryStream();
var emit = compiled.Emit(ms);             // emit ONCE
if (!emit.Success)
    throw new ScriptCompilationException(emit.Diagnostics);
ms.Position = 0;

var alc = new CollectibleLoadContext();   // isCollectible: true
var asm = alc.LoadFromStream(ms);
// ... invoke the delegate N times against ScriptGlobalVariables ...
alc.Unload();                             // dynamic assemblies reclaimed

5.4 Static state in transitive dependencies

The collectible load context guarantees that dynamically generated assemblies are reclaimed on unload. It does not, on its own, guarantee that references held by libraries outside the context are reclaimed. Several common .NET libraries register process-wide singletons or hold static caches that can pin objects across the boundary: HttpClient's default handler pool, Npgsql's connection pool, Confluent.Kafka's producer cache, OpenTelemetry tracer providers, logging providers registered through ILoggerFactory. When a Core or community provider's emitted code constructs one of these, the resulting object may be retained even after the load context unloads, and over thousands of iterations memory will grow.

The architecture treats this as a first-class concern with three rules. First, provider services are exposed to the script only through the global-context object, never through static fields the provider's own assembly owns. Second, provider-managed resources (connection pools, broker clients) live inside the engine's durable services container, are created and disposed by the engine's lifetime hooks rather than by the script, and are exposed to the script as already-initialised handles. Third, a memory-leak regression test runs in CI against the full transitive closure of every Core provider, not just against trivial scripts, so that a dependency-introduced leak is caught early rather than near the pilot. This third rule is named explicitly as a first-milestone deliverable because the cost of discovering a dependency-introduced leak late is much higher.

5.5 Dynamic assembly resolution for customer code

Advanced scenarios need the customer's own libraries — a proprietary serialisation format, a shared domain model, an internal client. The Roslyn engine accommodates this by populating the compilation through ScriptOptions.Default.AddReferences() and AddImports() before compilation. The customer points the platform at a set of DLLs or NuGet packages; the generated script can then construct and assert against those types directly. This is the mechanism that satisfies design goal five — customers extend behaviour without the platform team shipping a release — and it is also a security boundary, since arbitrary customer code now runs inside the test process. Section 11 returns to the sandboxing implications, and the assembly-graph hygiene rules in Section 5.6 govern how customer DLLs coexist with provider services without symbol clashes.

5.6 Assembly-graph hygiene

Customer DLLs, provider service types, and engine types all live in one process and contribute types to one script compilation. Three rules keep the resulting graph navigable rather than a source of late-binding bugs.

Reserved namespaces. Engine types live under Vouchfx.Engine.*; provider-supplied service types live under Vouchfx.Steps.* with the family and provider name suffixed (e.g. Vouchfx.Steps.DbAssert.Postgres). Customer DLLs must not declare types in these namespaces; if they do, the engine refuses to load them at suite start with a clear diagnostic, rather than silently shadowing platform types.

Customer DLLs share the script's load context. Customer-supplied assemblies are loaded into the same collectible AssemblyLoadContext as the generated script, not into the engine's default context. This means they are reclaimed alongside the script when the context unloads, and it means a customer DLL whose initialiser raises an exception fails the affected suite rather than poisoning the engine process.

Conflicting transitive dependencies fail fast and clearly. When a customer DLL transitively pulls in a different version of a package the engine or a provider already references (the classic Newtonsoft.Json or Confluent.Kafka mismatch), the resulting AssemblyLoadException or version-mismatch warning is surfaced at suite start, with the conflicting assembly identities and the recommended action (align versions, or rebuild the customer DLL against the engine's published binding contract). The default behaviour is to refuse the suite rather than to hope for the best at runtime.

5.7 Library and version commitments

The compiler and runtime layer depends on a small number of external libraries whose specific versions affect provider implementations. The platform commits to the following at v1.0.

Concern	Library and version	Why
Resilience policies (RETRY)	Polly v8 via Microsoft.Extensions.Resilience.	Polly v8's ResiliencePipeline is the modern API and is what every provider's RETRY emission targets. Polly v7 is incompatible and is not supported.
JSON parsing and serialisation	System.Text.Json.	The .NET 8 LTS default; consistent with Aspire and the rest of the platform's surface. Newtonsoft.Json is not used in engine code; customer DLLs that reference it work but the engine never returns Newtonsoft types to provider code.
JSONPath extraction	JsonPath.Net (the JsonEverything family).	Built on System.Text.Json.Nodes; the de facto modern JSONPath implementation in the .NET ecosystem. The engine pins a specific version and exposes it through the global context rather than re-exporting the library publicly.
XPath extraction	System.Xml.XPath.	Part of the framework; no external dependency.
YAML parsing	YamlDotNet.	The de facto choice; stable and feature-complete for the DSL's grammar.
JSON Schema validation	JsonSchema.Net (the JsonEverything family), draft 2020-12.	Aligns with the JSONPath choice; supports the if/then/else-with-const-discriminator composition pattern Section 13.6 specifies.
Container clients (Core providers)	Npgsql, Confluent.Kafka, MongoDB.Driver, StackExchange.Redis, NEST/OpenSearch.Client.	Each is the .NET-canonical client for its technology. Pinned to specific majors; provider authors target the same majors so the engine can guarantee one version per package across the closure.
Avro serialisation and schema governance (message publishing)	Confluent.SchemaRegistry, Confluent.SchemaRegistry.Serdes.Avro, Apache.Avro.	Version-locked in lockstep with Confluent.Kafka (2.14.2); the Serdes.Avro adapter integrates Avro publishing/consuming with the schema registry, and Apache.Avro is pinned to the exact transitive version Serdes.Avro requires to avoid downgrade warnings under strict compilation flags.

Table 5.1 — Library and version commitments at v1.0. The platform pins majors and selects minors to track upstream security patches; providers target the same majors so the engine's load context never serves two versions of the same package.

.NET runtime baseline. The platform targets .NET 8 LTS as its runtime, because most adopter organisations are on the current LTS and will not move to a non-LTS version of .NET to adopt a testing tool. This is the runtime the platform itself requires, not a constraint on the system under test: the system under test may be on any .NET version that runs in a container, while the platform's engine, CLI, and extension run on .NET 8 LTS.

Aspire version stance. Aspire's API surface is still moving across minor versions, and the platform team has been bitten by tracking previews. The platform pins to a known-stable Aspire minor version per engine release and follows Aspire forward at engine-minor cadence, not Aspire-minor cadence. Adopters therefore upgrade Aspire alongside the engine rather than independently of it. This trade-off buys adopters predictability at the cost of some Aspire-newer-feature availability, which is the right trade for a tool that must be stable enough to trust in CI.

6. The Testcontainer Cloud Execution Fabric

Running a full microservice topology locally is excellent for tight iteration, but a suite containing SQL Server, Elasticsearch, several Kafka brokers, and dozens of application containers will exhaust the CPU, memory, and disk I/O of an ordinary workstation. The cloud fabric solves this by splitting the work: the lightweight parts — the YAML parser and the Roslyn delegate runner — stay local, while the heavy container provisioning is offloaded to a high-performance cloud backend. The challenge is to make that offload completely transparent to the orchestration layer above it.

6.1 Remote Docker socket proxying

Transparency is achieved by convincing the local Aspire and Testcontainers processes that they are talking to a local Docker daemon when they are in fact talking to the internet. Two pieces of configuration accomplish this. On the local side, the runner injects settings into the developer's ~/.testcontainers.properties file: it sets docker.client.strategy to the environment-and-system-property provider strategy, and redefines docker.host to point at a secure TCP proxy provisioned by the SaaS backend, for example tcp://agent.saas.cloud:2375. On the cloud node, the Docker daemon's systemd unit is configured to bind a TCP socket alongside the local Unix socket. Container-provisioning API calls then traverse the internet to the remote daemon while every layer above believes the daemon is local.

# Cloud node — /etc/systemd/system/docker.service.d/override.conf
ExecStart=/usr/bin/dockerd --config-file /etc/docker/daemon.json \
          -H unix:///var/run/docker.sock -H fd:// -H tcp://0.0.0.0:2375

# Local node — ~/.testcontainers.properties
docker.client.strategy=org.testcontainers.dockerclient.\
    EnvironmentAndSystemPropertyClientProviderStrategy
docker.host=tcp://<cloud-agent-ip>:2375

On the local agent the connection is marshalled by a SocketsHttpHandler with a custom ConnectCallback, which can route over a UnixDomainSocketEndPoint for local execution or a TCP endpoint for remote execution — the same code path, a different endpoint, which is precisely how parity is preserved.

6.2 Encrypted tunnelling and reverse port forwarding

Offloading infrastructure creates a two-way networking problem. The local test script must reach cloud-hosted databases, and — harder — a cloud-hosted service must sometimes reach back to the local machine.

6.2.1 Outbound: reaching cloud infrastructure

Each cloud session runs inside an ephemeral, isolated Virtual Private Cloud sandbox, which guarantees multi-tenant data isolation. A persistent, multiplexed, encrypted SSH tunnel connects the local runner to that sandbox, so local C# assertions can query a cloud-hosted database as though it were on localhost.

6.2.2 Inbound: the webhook problem

The hard direction is inbound. When a cloud-hosted microservice finishes work and fires an asynchronous webhook, that HTTP POST must reach the test runner on a laptop sitting behind a corporate firewall. The fabric solves this with reverse SSH port forwarding — conceptually the ssh -R remote_port:localhost:local_port pattern. The SaaS agent opens a port on the cloud network that tunnels straight back to an ephemeral HTTP listener inside the local test process. The cloud container issues its POST to what it sees as a local port; the request is routed through the SSH tunnel and delivered to the CSX script that is waiting for the callback. No inbound firewall rule on the developer's network is required.

6.3 Resource reaping: the Ryuk sidecar

Offloaded infrastructure introduces a cost risk: a crashed pipeline can leave containers, networks, and volumes running and billing in the cloud. The fabric guards against this with the Ryuk resource reaper, a privileged Go sidecar deployed alongside the test instances. During provisioning, Testcontainers tags every network, volume, and container with cryptographic labels. Ryuk holds a TCP connection to the test runner and watches it continuously. If that connection drops — a pipeline crash, a network partition, a SIGKILL to the local process — Ryuk purges every labelled resource, so an abnormal termination cannot produce a runaway cloud bill.

6.3.1 Graceful degradation where privileged containers are banned

Some enterprise Kubernetes environments forbid privileged sidecars outright. For these, the fabric offers a degradation path: setting TESTCONTAINERS_RYUK_DISABLED=true bypasses the sidecar, and cleanup falls back to native .NET process shutdown hooks — AppDomain.ProcessExit — which issue container-termination calls through the Docker API before the process dies. This is less robust than Ryuk against a hard kill, and section 13 records that trade-off as an accepted risk in the enterprise topology.

7. Target Environments and Topological Adaptability

The platform's defining value proposition is that one test suite runs unchanged across three very different environments. The architecture delivers this by isolating every environment-specific decision inside the infrastructure fabric described in section 6; the layers above it are reconfigured, never rewritten. The three topologies below differ only in where containers run and how identity is established.

7.1 Local execution — the indie developer

In the local tier, aimed at solo developers and open-source contributors, everything runs on the workstation. The Aspire AppHost talks directly to the local Docker socket — unix:///var/run/docker.sock on Linux and macOS, or a named pipe on Windows — to start the databases, brokers, and application containers. The YAML runner executes in the same network namespace as its targets and enjoys sub-millisecond latency. The tier has no external dependency and functions fully offline, which makes it the natural on-ramp and the foundation of the free edition.

7.2 SaaS execution — small and agile teams

For teams that cannot justify dedicated local hardware capable of running dozens of microservices, the planned SaaS (Team) tier offloads container provisioning to the cloud fabric. Developers still author locally with full IDE support, but execution targets the remote backend. This tier is the natural fit for CI/CD pipelines, where lightweight runner agents that lack nested virtualisation cannot start containers themselves and instead connect to the SaaS platform to do the heavy lifting.

7.3 Enterprise execution — on-premises and SSO

Regulated sectors — finance, healthcare, defence — frequently cannot use multi-tenant SaaS because of data-residency and network-egress rules. The planned Enterprise tier therefore packages the cloud execution engine as a deployable artefact: a Kubernetes Helm chart or a self-hosted Docker architecture installed inside the customer's own VPC or VNet. The execution model is identical to the SaaS tier; only the location of the backend changes.

7.3.1 Identity as part of the execution path

The deep architectural difference in the enterprise tier is identity and access management. The platform integrates with SSO providers through SAML 2.0 and OpenID Connect, and that integration is not confined to a reporting dashboard — it reaches into the test-runner CLI and the VSCode plugin. A developer authenticates local tooling against the enterprise identity provider and receives a short-lived JSON Web Token. That token authorises the provisioning of large container topologies and carries Role-Based Access Control claims. The effect is that authorisation is evaluated at the moment infrastructure is requested: every spin-up is logged to a central audit trail, and a junior user can be prevented from launching a resource-intensive load test that might destabilise a shared staging cluster.

Dimension	Local (free core)	SaaS (planned Team tier)	Enterprise (planned)
Where containers run	Local Docker socket	Multi-tenant SaaS cloud	Customer-managed VPC / VNet
Network model	Single local namespace	SSH tunnel + reverse forwarding	SSH tunnel within customer network
Identity	None required	Platform account	Enterprise IdP via SAML / OIDC
Authorisation	n/a	Team membership	JWT-carried RBAC, audited
Resource limit	Workstation capacity	Metered compute credits	Customer-provided compute
Offline capable	Yes	No	Yes, within customer network

Table 7.1 — The three topologies differ only in the infrastructure fabric and identity layers.

8. Autonomous Agentic Automation

The architecture described so far still assumes a human authors the YAML. The agentic layer relaxes that assumption, turning the platform from a declarative tool into a closed-loop quality system that can plan, generate, execute, and repair tests within defined boundaries. It is built on the Microsoft Agent Framework (MAF), the open-source multi-agent orchestration framework Microsoft positions as the production successor to Semantic Kernel. The framework targets .NET 8 and later (and other languages) rather than being tied to a single .NET version, and parts of it are still maturing as of the time of writing; the architecture treats MAF as a stable enough host for the agent layer but acknowledges the implementation will track its evolution. The agentic layer is post-MVP and the team will validate MAF's then-current capability set before committing the layer's detailed design.

8.1 The three-agent topology

Rather than one monolithic agent, the platform uses three specialised agents, each with a narrow remit. The separation keeps each agent auditable and positions the platform for future tiered offerings.

Agent	Trigger	Responsibility
Planner	New or changed user stories, acceptance criteria, requirements docs.	Connects to issue trackers such as Jira or GitHub, analyses requirements, and formulates a structured test plan covering positive, negative, and boundary cases.
Generator	A completed plan from the Planner.	Translates planned scenarios into the platform's YAML DSL: constructs JSON payloads, formulates REST and Kafka invocations, and emits any CSX logic required — replacing manual authoring.
Healer	A test failure at runtime caused by drift.	Operates beside the Aspire orchestration layer; analyses the stack trace of a failure caused by a changed endpoint or schema, updates the YAML or CSX to resolve the drift, and re-executes to confirm the fix.

8.2 Graph-based workflows and human-in-the-loop gates

The three agents do not run as a simple loop. They are dependency-injected services collaborating over the Agent Framework's graph-based workflows, which give type-safe message routing between agents, explicit execution paths, and checkpointing. The graph model is what makes the Healer safe to deploy: before the Healer commits a modified baseline test back to source control, the workflow can route through a human-in-the-loop approval gate, so an engineer confirms the repair rather than trusting it blindly. The agentic layer's purpose is to remove the two structural failure modes of legacy automation — coverage debt, because the Planner and Generator keep coverage current, and the maintenance bottleneck, because the Healer absorbs drift.

8.3 A note on the authoring inversion this enables

It is worth recording a trajectory the architecture now makes possible, even though it is not a near-term deliverable. The provider model of Section 13 and the structured event stream of Section 14 together make the test suite legible to a machine in both directions: the Generator can emit a provider-typed step, and the event stream can describe exactly what a step observed. That symmetry opens a future in which the relationship between human and agent inverts — the agent drafts the first version of a test from observed traffic or from a captured production trace, and the human becomes a reviewer of generated tests rather than their author. This is deliberately out of scope for the initial release, where the human authors and the agent assists, but the architecture is intentionally not closing the door on it. Designing the provider contract and the event stream to be machine-writable as well as machine-readable is what keeps that inversion available as a later strategic option rather than a rewrite.

9. The Integration and Performance Testing LAB

Because the orchestration layer already knows how to provision arbitrary containers and discover their endpoints, extending the platform from functional testing into load and stress testing is a natural rather than a disruptive step. Load testing verifies that a distributed system stays resilient and performant under traffic spikes and hardware stress — a different question from functional correctness, but one the same machinery can answer.

9.1 Containerised load injection

Industry-standard load tools run natively as containers. The LAB leverages the Testcontainers k6 module: during a test cycle the Aspire AppHost starts a grafana/k6 container on the same Aspire network as the services under test, mounts the load script translated from the YAML DSL as a bind mount, and injects the dynamically discovered backend endpoints as environment variables. The notorious manual wiring of service discovery for localised load testing simply disappears, because discovery is already solved one layer down.

9.2 High-performance job orchestration

Coordinating thousands of virtual users and the telemetry they generate demands a careful concurrency model inside the engine. The LAB relies on System.Threading.Channels for internal coordination — a thread-safe, allocation-efficient producer/consumer model that avoids the thread-blocking overhead of older constructs such as BlockingCollection. It uses bounded channels, created with Channel.CreateBounded<T>(), so that when consumers fall behind, producers are naturally throttled. This backpressure is what prevents the engine from overloading itself while driving a heavy stress test.

9.3 Why the LAB has distinct economics

A functional test verifies a single data flow; a load test simulates thousands of concurrent users and generates large volumes of metrics, traces, and logs. That difference spikes CPU, memory, and network-egress consumption on the cloud backend far beyond what a functional run consumes. The architecture therefore meters LAB usage separately and precisely, scaling with the intensity of the load, so that the heavy computational and storage demands remain sustainable. The engineering point is that the LAB must emit accurate, per-session consumption telemetry; metering correctness is a functional requirement of this subsystem, not a downstream concern.

10. System Resilience and State Management

A platform whose job is to orchestrate volatile, asynchronous distributed systems must itself be resilient, because it operates in exactly the chaotic network conditions it is designed to test. Two problems dominate: flakiness from race conditions, and partitions between the local runner and the cloud fabric.

10.1 Race conditions and asynchronous event tracking

Integration tests are notoriously flaky because of unpredictable latency, variable container startup, and genuine race conditions. The architecture's answer is to lift retry and polling logic entirely out of the test author's hands and embed it in the engine. The DSL exposes a verification construct — written in YAML as VERIFYMODE: RETRY — and when a step uses it, the compiler generates a Polly resilience policy that performs bounded, exponential-backoff polling against the target broker or database. Because the engine owns the timing, a test behaves identically on a fast laptop and a heavily loaded shared CI runner. Authors never write Thread.Sleep() or hand-rolled while loops, which are the classic sources of both flakiness and wasted time.

10.2 Stateful transactions and variable capture

Real end-to-end scenarios are stateful. A test might create a user through a REST API, capture the generated user_id, and then assert that a Kafka event carrying that exact identifier was processed by a billing service and indexed into Elasticsearch. The engine threads this state through the mutable dictionary inside the ScriptGlobalVariables object. As the script executes steps in sequence, JSONPath or XPath extractors pull values out of HTTP responses and store them in the global state; later C# blocks reference those dynamically generated identifiers directly. State management is therefore not a special feature but a property of the same global-context bridge that section 5 introduced.

10.3 Infrastructure fault tolerance and auto-recovery

In the SaaS model, a network partition between the local environment and the cloud containers is inevitable rather than exceptional. The architecture treats a momentary drop as recoverable. If the connection to the remote Docker socket drops, the local client agent halts the CSX execution pipeline instead of failing the suite outright, and attempts auto-reconnect of the TCP socket forwarding. In parallel, the remote Ryuk sidecar is configured with a generous time-to-live heartbeat: when it loses contact with the client it does not reap immediately but waits out a grace period, giving the SSH tunnel time to re-establish. Only when that grace period definitively expires does Ryuk conclude the client has failed and sanitise the VPC. The result is that a brief internet blip produces a pause, not a false-positive failure — which is decisive for developer trust in the SaaS tier.

11. Security Architecture

Security in this platform is unusual because the platform deliberately compiles and runs arbitrary code — both generated CSX and customer-supplied DLLs — and provisions infrastructure on the customer's behalf. Security must therefore be designed into the execution path, not bolted on. This section consolidates the security-relevant decisions made elsewhere in the document, states the threat model the architecture defends against, and names the boundaries an implementation must enforce.

11.1 The threat model

Stating the threat model in one place serves two purposes. It gives security reviewers a single artifact to assess against; it also exposes the assumptions later subsections rely on, so that a reader can audit those assumptions rather than reconstruct them. The architecture defends against the threats below; threats explicitly out of scope are also listed because their absence is sometimes mistaken for an implicit defence.

Asset	Trust boundary	In-scope threat	Defence
Test source code and YAML	Customer's source control to engine	Tampered test file mis-executes the system under test.	Reproducibility envelope records content hashes of every YAML; structured event stream attests to what actually ran.
Customer's application secrets	YAML to runtime	Secret value leaks via report, log, captured-variable thread.	${secret:…} references resolved at step-execution time only; typed SecretString return that the renderer is contractually forbidden to log; redaction at source, not at sink (see 11.5).
Customer's application code	Engine to script's load context	A malicious customer DLL exfiltrates data from the test process.	Documented in 11.3 as inherent to the trust model: running tests from authors you do not trust is equivalent to running their arbitrary code. The architecture mitigates blast-radius, not the underlying authorisation.
Provider implementation	Community provider to engine	A malicious community provider runs hostile code in every install that uses it.	Two-tier governance (Core, Community) with the maintainer-awarded Vouched badge and published rubric (see VOUCHED_CHECKLIST.md in the hub); package signature verification at install time; security review at badge award; explicit warning for unbadged community providers.
Cloud fabric multi-tenancy	One customer session to another	Container, network, or data escape from one tenant's VPC to another's.	Ephemeral per-session VPC isolation; Ryuk-enforced teardown on session end; absence of long-lived shared state by design.
Cloud tunnel credentials	Local runner to cloud fabric	Static credential theft permits a long-lived attacker session.	Short-lived OIDC-federated credentials are the v1.0 default; the SSH tunnel is authenticated by those credentials, refreshed per session. Static long-lived keys are not the default mechanism.
Platform-issued telemetry	Customer machine to platform backend	Sensitive data exfiltration via telemetry.	Opt-in by default; strict allowlist of metric fields (see 11.6); no test contents, captured values, error messages, or system-under-test addresses; bounded retention; access-controlled inside the platform team.
The Healer agent's commit path	Agent suggestion to source control	Healer modifies code without meaningful human review and a hostile suggestion lands.	Future-tier, but the design choice is made here: Healer-authored commits are technically distinguishable from human commits (separate signing identity, mandatory CODEOWNERS review path), so the human-in-the-loop gate is enforced by tooling rather than by user discipline.

Explicitly out of scope. The platform does not defend against: a malicious developer with legitimate write access to test files and access to customer credentials (this is the same threat any test framework faces and is an authorisation concern at the customer organisation); supply-chain compromise of the upstream .NET runtime, Aspire, or Testcontainers (defence-in-depth via package signature verification, but not a primary defence); the customer's own network or cloud-provider compromise (out of the platform's control). The platform's protection of customer secrets in transit and at rest assumes the customer's machine itself is not already compromised; defending against rootkit-level attackers on the developer's laptop is not the platform's job.

Vulnerability reporting. The published vulnerability-disclosure policy, supported versions, coordinated-disclosure SLA, and in/out-scope guidance live in SECURITY.md at the repository root.

Release signing and provenance. All releases are signed keylessly using Sigstore (OIDC-federated, no long-lived key management) and include verifiable SLSA build provenance. Consumers verify release artefacts with gh attestation verify or cosign verify-blob. The signing pipeline is defined in .github/workflows/release.yml and activates when binary packaging ships; see that file for the keyless model and verification procedure.

11.2 Identity and authorisation

In the enterprise topology, identity flows from the customer's SAML 2.0 or OIDC provider into short-lived JWTs that the CLI and IDE plugin obtain and present. Authorisation is evaluated when infrastructure is requested, so RBAC claims gate provisioning rather than merely gate a dashboard login. Every provisioning event is written to a central audit trail. The SaaS topology uses platform-managed accounts and team membership; the local topology requires no identity because no shared resource is involved.

11.3 Multi-tenant isolation

Each cloud session runs in its own ephemeral, isolated VPC sandbox, which is the primary boundary preventing one tenant from observing another's containers, networks, or data. Sessions are short-lived by design, and Ryuk guarantees that their resources are destroyed on termination, so isolation is reinforced by the absence of long-lived shared state.

11.4 The dynamic-code boundary and the customer-DLL trust statement

Plain statement of the trust model

Running tests authored by parties you do not trust is equivalent to running their arbitrary code on your system. The platform is, by design, a Roslyn-based test execution environment with a customer-DLL extension surface (script.csharp step and ScriptOptions.AddReferences). The architecture limits blast radius (collectible load context, reserved namespaces, least-privilege process) but does not sandbox malicious test authors away from the host. Adopt the platform with the same authorisation discipline you apply to any other test framework: trust the people who write tests, or do not let them run unreviewed.

With that stated plainly, the architecture's defensive posture is as follows. The most security-sensitive surface is the Roslyn execution layer. Generated CSX is constructed by the platform and is comparatively trustworthy; customer DLLs added through AddReferences() are not. An implementation must treat the test process as a place where untrusted code runs and constrain it accordingly: the collectible AssemblyLoadContext already bounds the code's lifetime, and the process should additionally run with least-privilege credentials, restricted filesystem access, and no ambient access to platform secrets. The global-context object should expose only the clients and connection strings a test legitimately needs. This is an area where the blueprint states the requirement and defers the detailed sandboxing design to a dedicated security review, recorded as a risk in section 18.

11.5 Platform-generated session credentials and transport

This subsection is about the credentials the platform itself generates and transports — the per-session tokens and connection strings the cloud fabric mints for a runner to talk to a remote topology, and the SSH tunnel that protects them. It is distinct from user-supplied application secrets — database passwords, API keys, OAuth client secrets — which authors reference from their YAML through the ${secret:…} mechanism described in Section 17. The two concerns are different: platform-generated session credentials are the runner's identity for talking to the platform; user-supplied secrets are the application's identity for talking to the system under test. Both exist; they do not overlap.

Tunnel authentication is OIDC-federated short-lived credentials at v1.0 of the cloud tier. A runner authenticates to the platform's identity service (federated to the customer's identity provider where the Enterprise tier is in use) and receives a short-lived token (typical lifetime: one hour, refreshable) that authorises the SSH tunnel for that session. Static long-lived SSH keys are explicitly not the default mechanism, because they are a credential-theft target whose compromise persists indefinitely; the short-lived token model is the same one cloud providers have standardised on (AWS STS, Azure managed identities, Google Cloud Workload Identity Federation) and is the modern minimum bar. Customers needing a fallback for environments where federation is not yet available may use a static credential, with the documented warning that the trade-off is theirs.

Connection strings and tokens of the first kind are never persisted in test artefacts. All traffic between the local runner and the cloud fabric travels inside the encrypted SSH tunnel; the Docker TCP socket is never exposed without that tunnel in front of it. In the enterprise topology, both kinds of secret remain entirely within the customer's network because the backend itself runs there.

11.6 Secret redaction at the source, not at the sink

The ${secret:…} mechanism in Section 17 returns a typed SecretString wrapper from Vars.Secrets.Resolve, not a bare string. The SecretString type implements only operations a credential legitimately needs (passing into an HTTP header, a database connection string, a signing key for HMAC) and explicitly does not implement ToString() or IFormattable in a way that returns the underlying value; attempts to log it via ILogger or to print it through Console.WriteLine yield the literal ${secret:…} reference string, never the resolved value. This is redaction at the source, and it is more reliable than the earlier string-match-based redaction at the renderer: a string-match approach fails on base64-encoded credentials, HMAC-signed credentials whose signature is what appears in logs rather than the secret itself, and partially-logged credentials (the first eight characters of an API key). The typed wrapper handles all three cases by construction.

Renderers that consume the structured event stream still apply a string-match redaction pass as defence-in-depth, because some leaks happen through provider-emitted code that bypasses the typed accessor (a provider that does the unsafe thing intentionally). The two layers together are stronger than either alone, but the source-typed redaction is the primary defence.

11.7 Telemetry and the platform's privacy stance

A test platform generates an unusual amount of information about its users' systems — service names, test contents, captured variable values, dependency types, container images. The architecture takes a deliberately restrictive stance on what leaves the user's machine, because the trust the platform needs from a senior engineer is incompatible with default-on harvesting.

The default is no telemetry. At first run the tool prints a notice describing what telemetry would be collected and the command to enable it; nothing is collected until the user opts in explicitly. When telemetry is enabled, the tool reports only structural counts and timings: number of runs, scenarios per run, verdicts by category, steps by family and provider, suite startup time, time-to-first-passing-test, an anonymous installation identifier, the platform and tool version. The tool never reports test file contents, captured variable values, secret references or resolved values, error messages, system-under-test addresses, container image names, or any data that originated from the system under test. A test file may declare itself non-telemetered with a metadata flag, in which case no run-level telemetry is reported from that file regardless of the global setting.

Access, residency, and retention. Telemetry data, when present, flows to a backend the platform team operates in EU-region infrastructure (the working assumption based on the team's location; the precise region commitment is published before any Enterprise pilot involves real data). Access inside the platform team is limited to a named set of engineers and is auditable. Data is retained for ninety days; disabling telemetry deletes the installation identifier from the backend within thirty days. Telemetry data is never shared with third parties. This commitment is encoded in the v1.0 release manifest and surfaced to the user at first run; the detailed v1.0 telemetry policy is published in docs/telemetry.md.

12. Observability and Operability

A test platform has a special obligation that ordinary software does not: it must let a user distinguish a genuine product defect from a failure of the test environment itself. If the two are conflated, every flaky environment erodes trust in real findings. Observability is therefore a first-class architectural concern.

12.1 The verdict taxonomy

Every step and every scenario in the platform terminates with exactly one of four verdicts. This subsection is the single authoritative definition; the reporting layer (Section 14) cites it as the basis for its rendering rules, and the rest of this document refers back here. The taxonomy must be applied deliberately rather than collapsing everything into pass or fail, because the four cases require fundamentally different responses from the reader.

Outcome	Meaning	Where it originates
Pass	All assertions satisfied within their verification windows.	Execution layer.
Fail	An assertion was evaluated and did not hold — a genuine defect in the system under test.	Execution layer.
Environment error	The step could not be evaluated because surrounding infrastructure failed: a container image could not be pulled (registry unreachable, authentication failure, rate-limited, or image not found), a container never became healthy, a dependency could not be provisioned, a broker rejected the connection, a tunnel collapsed, or a seed fixture failed to apply. Image-pull failures are further classified by authentication outcome: "unauthenticated" (401), "access-denied" (403), "rate-limited" (429), "anonymous" (non-auth failure), or null (registry unreachable).	Orchestration / fabric layers.
Inconclusive	Evaluation began but no verdict could be reached. Three sub-cases share this label: (a) a timeout fired on a step waiting for an external system that never responded; (b) a network partition outlasted the engine's grace window; (c) a step was skipped because an earlier step's capture was unmet and the inputs for evaluating this step therefore did not exist.	Execution layer (a, c) and resilience layer (b).

Only Fail should break a CI build by default. Environment errors and inconclusive results should be surfaced distinctly so that a team can fix infrastructure or repair an upstream step without distrusting their assertions. The reporting layer renders each verdict with a distinct colour and counts them separately; the precise visual treatment is specified in Section 14.

12.2 Telemetry signals

Each layer emits structured telemetry. The orchestration layer reports container startup durations and health-gate timings. The compilation layer reports compile time and, critically, memory before and after each load-context unload, so that a regression in reclamation is caught early. The execution layer emits per-step latency and per-assertion outcomes. The cloud fabric reports tunnel stability, reconnection events, and — for the LAB — the per-session consumption metrics that billing depends on. Correlating these signals lets an operator answer the question every test platform must answer: was it the code, or was it us?

12.3 Consumption accounting is measured from day one

A specific commitment follows from the telemetry design, and it is worth stating explicitly because it is cheap to honour early and expensive to retrofit. The engine measures per-resource consumption — container-minutes by image, suite wall-clock duration, queue time before a topology becomes ready, peak concurrent resources, and data volume moved through the fabric — from the very first release. Capturing the raw consumption signal as a functional requirement of the execution fabric keeps every future capacity and cost analysis supportable by data the engine already collects. The alternative — adding consumption measurement only once it is needed — means discovering that the historical data required to validate any analysis was never recorded. Measuring from day one is therefore the deliberate sequence.

13. The Provider Plugin Architecture

Step types in the DSL — http, db-assert, mq-publish — name what is being done at the level of intent, not at the level of implementation. There is no universal way to assert against a database: PostgreSQL gives the author JSONB, arrays, and RETURNING clauses; SQL Server gives MERGE statements and a CLR type system; Oracle has its own NULL semantics and PL/SQL extensions; MongoDB is not SQL at all; Redis is not even relational. The same fracture applies to messaging — Kafka offsets are nothing like RabbitMQ acknowledgements, which are nothing like SQS visibility timeouts — and to synchronous remote calls, where REST, SOAP, and gRPC share a transport and almost nothing else. A platform that fixed one implementation behind each step type would either constrain authors to a lowest-common-denominator subset or grow a tangle of switch statements inside the engine. Neither outcome is acceptable for a tool that intends to last.

This section defines the architecture's answer: a provider model that lets the DSL grow new step kinds through additional source code rather than through engine changes. Each step type is a family — db-assert, mq-publish, rpc, http, script — and each family is implemented by one or more providers. Providers are first-class participants in the compilation pipeline, contribute schema fragments to the editor experience, declare the orchestration resources they need, and live alongside the engine as Apache-licensed source code that any team in the community can extend by pull request. There is no runtime plugin loader, no dynamic assembly resolution, and no sandboxing surface introduced by this model: every provider in a given installation has been compiled into the engine binary that runs it.

13.1 Why the DSL must be extensible at the source level

Three forces push the decision toward extensibility, and they all point to the same answer.

The first is that a microservices estate spans many technologies. A team that runs PostgreSQL for its transactional store, MongoDB for projections, Redis for caches, Kafka for events, and SQS for outbound notifications cannot test its system end to end through a single db-assert or mq-publish verb without one of those verbs misrepresenting the technology underneath. Each store has its own query semantics, its own connection model, and its own way of signalling success — and a faithful test must respect those differences rather than paper over them.

The second is that the platform team cannot, and should not, own implementations for every technology in use. The list grows with the industry: ScyllaDB, DuckDB, CockroachDB, NATS, Pulsar, ClickHouse, OpenSearch — and that is only the public list, not the long tail of in-house adapters every large engineering organisation accumulates. Centralising implementation makes the platform a bottleneck and the project a backlog of feature requests.

The third is that the customers most likely to extend the platform — engineers in the .NET community — already think in terms of NuGet packages and pull requests. The natural collaboration model for them is to fork, implement, and contribute back, not to write a dynamic plugin against a runtime API. The architecture honours that by making provider extension a source-level activity: a contributor adds a project to the solution, implements a small set of interfaces, opens a pull request, and once merged the new step kind is available the next time the engine is built.

The compile-time plugin principle

Providers are pulled in by source-level reference, not by runtime discovery. A contributor's pull request adds a project to the solution; the engine references it; the new step kind exists from the next build forward. There is no separate plugin host, no dynamic assembly loading, no sandbox: every provider that runs in production has been seen by the C# compiler that built the engine.

13.2 Step families and the provider naming convention

The DSL's type field carries two pieces of information in one dotted name: the family — the abstract operation being performed — and the provider — the concrete technology that implements it. The form is <family>.<provider>, read aloud as “db-assert by postgres” or “mq-publish by kafka”. The convention is borrowed from Terraform, kubectl, and Ansible because it has proved durable there: it is short enough to read fluently, structured enough to filter and discover, and stable across decades of language evolution.

Family	Meaning	Representative providers
http	Issue an HTTP request and assert on the response.	http.rest, http.soap, http.graphql
rpc	Make a typed remote procedure call.	rpc.grpc
mq-publish	Produce a message onto a broker.	mq-publish.kafka, mq-publish.rabbitmq, mq-publish.servicebus, mq-publish.sqs
mq-expect	Consume from a broker and assert on the message.	mq-expect.kafka, mq-expect.rabbitmq, mq-expect.servicebus, mq-expect.sqs
db-assert	Query a data store and assert on the result.	db-assert.postgres, db-assert.sqlserver, db-assert.oracle, db-assert.mysql, db-assert.mongodb, db-assert.elasticsearch, db-assert.redis
webhook-listen	Open a listener and assert on the inbound call.	webhook-listen.http
script	Execute a code block in the chosen language.	script.csharp

Table 13.1 — Families, the operation each one names, and providers that implement them. The permanently free Apache-2.0 core ships an initial set of twenty-five Core providers; the community is expected to grow the right-hand column over time.

There is no bare-family shorthand: the author always writes the dotted name — type: http.rest, never a bare type: http — even for a family with only one registered provider today. This was a deliberate subtractive change made pre-v1.0, while the schema was still unpublished: a family gaining a second provider later must never silently change, or invalidate, the meaning of a step type an existing file already uses. Tooling renders the same canonical dotted name in suggestions and error messages, so a file is always unambiguous regardless of which team wrote it or reads it back.

13.3 The provider contract

A provider implementation is a small set of C# types that declare the family-and-provider name they implement, supply a JSON Schema fragment for their YAML shape, bind a YAML node into a typed model, validate that model semantically against the project, declare any orchestration resources they need, and emit the C# fragment that runs the step. The interfaces below define that contract; a provider implements the ones that apply to it and ignores the rest.

// Identity
public sealed record StepKindId(string Family, string Provider);

public sealed record ProviderMetadata(
    string   Version,           // SemVer; the provider's own version
    string   MinEngineVersion,
    string   License,           // SPDX identifier; Apache-2.0 for the core OSS layer
    string[] Authors);

// A provider declares which step kind it implements.
public interface IStepProvider
{
    StepKindId       Kind     { get; }
    ProviderMetadata Metadata { get; }
}

// Marker interface for the typed YAML model produced by binding.
public interface IStepModel { }

// Parsing and schema. The fragment is merged into the unified JSON Schema
// at engine startup, so the VSCode extension picks it up automatically.
public interface IStepBinder<TModel> where TModel : IStepModel
{
    JsonSchemaFragment SchemaFragment { get; }
    TModel             Bind(YamlNode node, IBindingContext ctx);
}

// Semantic checks: target exists, dependency type matches, captured vars
// referenced earlier in the file actually exist, contracts agree.
public interface IStepValidator<TModel> where TModel : IStepModel
{
    ValidationResult Validate(TModel model, IProjectContext ctx);
}

// CSX emission: returns the C# fragment that will be spliced into the
// generated script and compiled once into a collectible load context.
public interface IStepCompiler<TModel> where TModel : IStepModel
{
    CsxFragment Emit(TModel model, ICompileContext ctx);
}

// Optional: declare orchestration resources the step needs at runtime.
public interface IResourceContributor<TModel> where TModel : IStepModel
{
    IEnumerable<ResourceRequirement> Resources(TModel model);
}

// Optional: declare services to be exposed through the script's
// strongly-typed ScriptGlobalVariables host object.
public interface IRuntimeServiceContributor
{
    IEnumerable<ServiceDescriptor> Services();
}

Two design points are worth underlining. First, the contract is split into binding, validation, and emission so that each can be tested in isolation: a provider author writes a binder test, a validator test, and a compiler test rather than a single monolithic integration test. Second, the contract is deliberately strongly typed — TModel is a real C# class with named properties, not a Dictionary<string, object> — so that the engine and the IDE can see the provider's intent and the emitter can produce strongly typed CSX. The cost is a class per provider; the payoff is that mistakes are caught at compile time inside the engine, not at runtime inside a test suite. IRuntimeServiceContributor is the one non-generic interface in the set, by design: a provider's service descriptor is independent of any specific step instance, so a TModel parameter would be unused noise.

IProjectContext and ICompileContext are engine-supplied, provider-consumed contexts — the engine is their sole implementor, so adding a member to either is non-breaking for providers (unlike the provider-implemented surface above, which is frozen and evolves only via new optional interfaces, §"Provider model"). Both now carry SuiteDirectory: the base directory relative file-path step fields (e.g. script.csharp's file) are resolved against, mirroring the directory environment.seed already resolves fixture paths against. DeclaredDependencies (IProjectContext) and Captures/CaptureExprs (ICompileContext) were added the same way in earlier sprints.

13.3.1 The CsxFragment composition contract

The engine assembles a single CSX script by splicing together the CsxFragment returned by every provider in the suite. Because CSX (Roslyn scripting) is not full C# — it has only top-level statements, restricted using placement, no top-level namespaces, and limited partial-class support — fragment composition needs a discipline that prevents one provider's emission from breaking another's. Without that discipline, the symptoms are confusing compile errors that look like platform bugs but are actually provider-collision bugs. The contract below is what every provider's IStepCompiler.Emit must respect.

A CsxFragment returned to the engine has three parts, each held in a separate field so that the engine can compose them deterministically rather than parsing strings.

public sealed record CsxFragment(
    IReadOnlyList<string> RequiredUsings,   // e.g. "Npgsql", "System.Text.Json"
    IReadOnlyList<string> RequiredHelpers,  // nested static classes, namespaced
    string                StatementBlock);  // a single { ... } block

Using directives are returned as a list of namespace strings, not as inline using lines in the statement block. The engine collects every provider's required usings, deduplicates them, and emits a single ordered block of using directives at the top of the composed script. Providers therefore never write using inside their statement block.

Helper types are returned as the full source of nested static classes whose names are prefixed with the provider's family-and-provider identifier (e.g. DbAssertPostgres_Helpers). The engine emits these helpers at script scope, after the usings and before any statement block. Providers do not declare top-level usings, top-level methods, top-level classes outside this convention, or top-level namespaces — doing so produces a CSX compile error the engine maps to a provider-authoring diagnostic with a clear message.

Statement blocks are returned as a single brace-enclosed C# statement block. The brace scope guarantees that local variables declared inside one provider's block do not collide with locals in another's: every step emits its own scope. Within the block, the provider may reference the shared Vars global context, any service the engine exposed through ScriptGlobalVariables, and any helper the provider declared in RequiredHelpers. The provider may not assume the existence of variables declared by other providers; cross-step state is mediated exclusively through Vars.

Two CSX-body constraints providers must respect, both discovered during an early provider-contract spike and recorded here so future authors do not rediscover them. First, the C# 8 using var declaration form is not legal inside a Roslyn script body — it produces a parse error in CSharpScript.Create even when the host runtime supports it. Providers must instead declare disposable resources as plain var and call .Dispose() explicitly at the end of the block, or wrap the resource in a try / finally. Functionally the difference is a few extra lines; the constraint is a property of the Roslyn scripting pipeline, not of the language version. Second, step identifiers in the YAML may contain hyphens, but the emitted CSX uses them as the suffix of locally-scoped variable names where hyphens are not legal C# identifier characters. Providers must sanitise the step id (replace - with _) before splicing it into emitted variable names. The engine's helper CsxFragment.SanitiseId(string) is provided for this purpose and should be used in preference to ad-hoc replacement.

The per-step cancellation convention (step timeout enforcement). The assembler wraps every provider statement block in a per-step scope that guarantees two locals: __stepCt_<safeId> (a CancellationToken — None when the step declares no timeout; armed with the declared budget for an IMMEDIATE step; the engine's attempt token under RETRY) and __stepBudgetGoverned_<safeId> (true only for an IMMEDIATE step with a declared timeout). A conforming provider passes the token into every awaited client call that accepts one, inserts catch (System.OperationCanceledException) when (__stepCt_<safeId>.IsCancellationRequested) { throw; } ahead of its own generic catch so a budget cut escapes provider error handling, and — where its emitted body sets a hard-coded transport timeout — lifts that bound to infinite when __stepBudgetGoverned_<safeId> is true (the declared budget is then the sole governing bound, so a budget longer than the convention is honoured; the SQL database providers likewise lift their command timeout, while providers with no such hard-coded convention rely on the token plus late supersession and discard the flag explicitly). Enforcement is entirely cooperative: the engine never abandons a running body (a zombie task mutating the shared Vars dictionary or outliving the collectible AssemblyLoadContext would violate the §5 memory model). A provider that ignores the token still gets deterministic late enforcement — an over-budget outcome is superseded by Inconclusive (step-timeout) — but its in-flight client call will overrun the budget rather than being cut, so observing the token is part of the contract community providers are expected to meet.

Code generation in the emitter: use C# 11 raw-string interpolation with double dollars. Provider emit code constructs CSX fragment bodies as strings that contain literal braces (the script's own { and } code-block delimiters) and interpolation holes (step-specific identifiers the emitter fills in). The canonical idiom is the $$"""…""" double-dollar raw string introduced in C# 11: with two leading dollars, {{ … }} denotes a literal-brace pair in the emitted code and {id} denotes a single-brace interpolation hole the C# compiler fills at emit time. The single-dollar form ($"""…""") inverts these meanings and produces compile errors as soon as the body contains any C# block, which it always does. Every Core provider's emitter uses the double-dollar form; the providers repository's worked-example provider demonstrates the pattern, and the integration-test fixture every Vouched provider passes includes a fragment-emission test that catches deviations.

A worked composition illustrates the contract. Two providers — db-assert.postgres and mq-publish.kafka — each return a fragment; the engine composes them into a single script as follows.

// Engine-composed CSX (illustrative)
// Usings: collected from every provider's RequiredUsings, deduplicated.
using System.Threading.Tasks;
using Npgsql;
using Confluent.Kafka;

// Helpers: collected from every provider's RequiredHelpers.
static class DbAssertPostgres_Helpers { /* provider-supplied */ }
static class MqPublishKafka_Helpers    { /* provider-supplied */ }

// Statement blocks: one per step, each in its own brace scope.
// Note the explicit Dispose() rather than 'using var' — the latter
// is not legal inside a Roslyn script body. Step ids are sanitised:
// 'orders-db' becomes the variable suffix 'orders_db'.
// Step 1 from db-assert.postgres (orders-db):
{
    var conn_orders_db = new NpgsqlConnection(Vars.ConnString("orders-db"));
    try
    {
        /* ... open, query, assert, capture verdict ... */
    }
    finally { conn_orders_db.Dispose(); }
}

// Step 2 from mq-publish.kafka (events):
{
    var producer_events = new ProducerBuilder<string, string>(
        Vars.KafkaConfig("events")).Build();
    try
    {
        /* ... publish, capture verdict ... */
    }
    finally { producer_events.Dispose(); }
}

This convention makes provider composition a tractable engineering problem rather than a string-splicing minefield. The CONTRIBUTING.md in the providers repository repeats the contract concretely, and the integration-test fixture every Vouched provider passes includes a multi-provider composition test that catches deviations before merge.

13.4 The compilation lifecycle of a step

With the contract in hand, the engine's existing compilation pipeline (section 5) extends naturally. Every step in a parsed YAML file flows through the same provider-mediated stages.

Stage	What happens	Provider role
Resolve	The compiler reads the step's type field, splits it on the dot into family and provider, and looks the result up in the registry. There is no bare-family resolution: a non-dotted type is rejected at AST-build time, and all step types are written in the fully-qualified family.provider form.	Identified by StepKindId.
Bind	The provider's binder converts the YAML node into a typed model and validates it structurally against the provider's JSON Schema fragment.	IStepBinder.
Validate	The provider's validator checks semantic correctness against the wider project: the target dependency exists, captured variables are referenced consistently, the verification mode is permitted for this kind.	IStepValidator.
Plan resources	The compiler aggregates each step's resource requirements into the orchestration plan, so that every container, port, and environment variable needed by the suite is known before Aspire starts.	IResourceContributor.
Plan services	The compiler collects the runtime service descriptors so that ScriptGlobalVariables can expose the typed clients the emitted CSX will use.	IRuntimeServiceContributor.
Emit	The provider's compiler returns a C# fragment that performs the step against the runtime services. Fragments are spliced together into the suite's CSX and compiled once into a collectible load context, exactly as section 5 prescribes.	IStepCompiler.

The crucial observation is that the existing memory-safety guarantees of section 5 — compile-once, collectible load context, explicit unload — do not weaken when providers are introduced. Providers contribute CSX text, not dynamic IL; the resulting assembly is owned by the same load context the rest of the engine creates and reclaims. A buggy provider can therefore emit a slow test or an incorrect assertion, but it cannot, by construction, defeat the memory model.

13.5 Resource contribution to the orchestrator

Many providers need infrastructure to operate against. A db-assert.postgres step needs a PostgreSQL instance reachable at a known address; a mq-publish.rabbitmq step needs a broker with a known set of exchanges. The IResourceContributor interface lets a provider describe those needs declaratively, and the orchestrator (section 4) is what fulfils them.

Two modes are supported and the choice is the author's, not the provider's.

In explicit-dependency mode, the test file declares the dependency in its environment.dependencies section — for example orders-db: { type: postgres, version: "16" } — and the provider attaches to it. This is the form most projects use, because it makes the topology visible in the file itself and lets multiple steps share a single instance.

In ephemeral mode, the step omits the target field and the provider asks the orchestrator to spin up a private instance for the suite. This is useful for isolated assertions that should not share state with the application under test. The orchestrator still owns lifetime and health-gating, so ephemeral resources participate in the same start, ready, and teardown lifecycle as declared dependencies.

Whichever mode the author chooses, the provider does not call Docker itself, does not maintain a connection pool, and does not handle teardown. Those responsibilities remain with the orchestration layer, and the provider's only output is the declarative description of what it needs.

13.6 Schema composition and IDE integration

The single JSON Schema described in the DSL specification is, in fact, assembled. At engine startup the registry walks the registered providers, collects each provider's schema fragment, and merges them into a unified schema. The composition uses an if / then / else chain with a const-valued discriminator on the type field, rather than a flat oneOf array. The two are semantically equivalent but the former is dramatically faster to validate: a oneOf of fifty provider fragments forces the validator to try every branch on every node, where the const-discriminator pattern selects exactly one branch on the first comparison. Validation performance matters because the VSCode extension validates on every keystroke; a 50-provider catalog must not produce sluggish autocomplete.

The discriminator clauses are emitted in deterministic (type-key-sorted) order so the composed schema is byte-stable and snapshottable for change tracking. For the v1 engine series, the v1 schema — composed from all twenty-five Core providers' fragments — is frozen by a golden-file CI gate that fails if the composed JSON drifts, ensuring any schema change is deliberate and reviewed.

The engine and the VSCode extension use JsonSchema.Net (the JsonEverything family) against the JSON Schema draft 2020-12 vocabulary. The choice is consistent with the JSONPath library named in Section 5.7 and is the modern .NET-canonical pair. The unified schema is exposed at a stable URL the editor consumes; nothing in the editor's pipeline changes when a new provider is added.

The result is that the editor experience extends naturally as providers are added. Autocompleting type: db-assert. offers every registered db-assert.* provider; selecting db-assert.postgres narrows the remaining fields to the Postgres-specific schema; hover documentation comes from the provider's fragment; “Go to definition” on the type field navigates to the provider's source. The VSCode extension does not need to know which providers exist — it asks the unified schema, and the schema answers.

This is also how the documentation generator works: it reads the unified schema and produces a reference for every step kind, including community providers, without further effort from the platform team. A provider's README becomes the long-form prose, but the field reference and examples flow from the schema fragment automatically.

13.7 The provider registry and source-level discovery

Discovery is deliberately static. At engine startup the registry uses reflection over the set of referenced assemblies to find types tagged with [StepProvider] or implementing IProviderModule, instantiates them, and indexes them by StepKindId. The result is a frozen, immutable registry for the lifetime of the process. There is no late binding, no plugin loader, and no path-scanning.

// Convenience attribute for single-provider assemblies.
[AttributeUsage(AttributeTargets.Class)]
public sealed class StepProviderAttribute : Attribute { }

[StepProvider]
public sealed class DbAssertPostgresProvider :
    IStepProvider,
    IStepBinder   <DbAssertPostgresModel>,
    IStepValidator<DbAssertPostgresModel>,
    IStepCompiler <DbAssertPostgresModel>,
    IResourceContributor<DbAssertPostgresModel>
{
    public StepKindId       Kind     => new("db-assert", "postgres");
    public ProviderMetadata Metadata => new(
        Version:          "1.0.0",
        MinEngineVersion: "1.0.0",
        License:          "Apache-2.0",
        Authors:          new[] { "acme-platform" });
    // ... Bind, Validate, Emit, Resources implementations ...
}

// Multi-provider module: a single assembly registers several providers.
public interface IProviderModule
{
    IEnumerable<IStepProvider> Providers();
}

Because discovery is reflective rather than configured, adding a provider really is a matter of writing the class and adding the assembly reference. There is no central registry file to edit, no version-control merge conflict on a registry list, and no risk of forgetting to register a new class.

13.8 Versioning, capability declaration, and deprecation

Each provider declares a SemVer version through its ProviderMetadata and the minimum engine version it requires. At startup the registry refuses to load providers requiring a newer engine, with an actionable error rather than a silent failure. Providers can also declare capability flags — for instance, whether their family supports verifyMode: RETRY, whether they implement ephemeral mode, or whether they handle Avro and schema-registry binding — so the compiler can validate the suite against what the installed providers actually support.

Deprecation is first-class. A provider can register a step kind with a Deprecated flag and a pointer to its replacement; the compiler emits a warning at every use, and the documentation generator marks the kind as deprecated. This is how the language evolves without a hard fork: an old step kind is supported alongside its replacement for a deprecation window, then removed.

13.8.1 The engine-side compatibility commitment

The provider contract that providers depend on — the interfaces in section 13.3, the context types they receive, the [StepProvider] attribute, the JSON Schema fragment format, the CsxFragment composition contract — is frozen for the v1.x engine series. Within v1.x, the engine may add optional capabilities through extension interfaces (for example, IStepProviderV1_1 with default-implemented members, or new optional context properties) but does not modify existing interfaces, rename existing types, or change the meaning of existing methods. Providers built against v1.0 therefore continue to compile and run against every v1.x engine release without modification.

Breaking changes to the contract appear only in a v2.x engine series. A v2 engine declares the v1 contract dropped or evolved, and providers targeting v1 either continue to work in a v1-compatibility mode or are explicitly required to migrate. Providers declare which engine major they target through their MinEngineVersion, which the registry compares against the engine's own version at startup; an incompatibility produces an actionable error rather than a silent load. This commitment is essential for the open-source community pathway: external contributors will not maintain providers if the platform reserves the right to break their work on every release. Freezing the v1 contract is the minimum credible offer the platform owes its community.

This freeze is now enforced by a golden-file CI gate (SdkContractFreezeTests) that reflects over the entire Vouchfx.Sdk public surface and asserts it is byte-for-byte identical to the committed golden, failing if the contract drifts. The extension mechanism is demonstrated and tested through OptionalExtensionInterfaceTests, proving that new optional capabilities (such as IStepDiffRenderer and IHostResourceContributor, both added this way during v1 development) can be added by implementing new interfaces outside the frozen core, with no change to any v1 interface. This pattern — extend via a NEW optional interface, never mutate a v1 interface — is the sole mechanism for evolution within v1.x.

13.9 Open-source governance: the two provider tiers and the Vouched badge

The provider model is most valuable when the community contributes to it freely, and the architecture makes that the default rather than the exception. Two tiers exist, each with its own quality bar, owner, and integration path.

Tier	What it contains	Quality bar	Where it lives
Core	The set shipped with the engine — twenty-five providers across eleven families (http.rest, http.soap; db-assert for Postgres, MySQL, SQL Server, MongoDB and DynamoDB; mq-publish/mq-expect for Kafka, RabbitMQ, NATS, Azure Service Bus and Redis Streams; cache-assert for Redis and Elasticsearch; mail-expect.smtp; webhook-listen.http; metrics-assert.prometheus; storage-assert.s3; trace-expect.otlp; script.csharp).	Same SLAs as the engine; supported by the platform team; covered by the engine's own integration test matrix.	Main repository, src/Providers/Core.
Community	Community-authored or externally maintained providers, including company-internal adapters, experimental kinds, and providers maintained outside the main repository.	Author's responsibility; not gated by core maintainers on merge; opt-in installation.	Community provider hub (vouchfx-providers), listed in the registry index; hosted under community/<provider-id>/ in the hub or externally maintained.

Both tiers use the Apache 2.0 licence, so contributions can move between tiers — a Community provider may earn the Vouched badge (see below), and a Vouched provider may be promoted to Core if it becomes universally useful — without IP friction. The CONTRIBUTING.md document in the providers repository defines the precise requirements: a worked YAML example, a README, an integration test against the matrix, a passing security checklist for any provider that handles credentials, and an acknowledgement of the licence.

The Vouched badge: optional endorsement for community providers. After listing in the hub's registry, a community provider may request the Vouched badge by demonstrating compliance with a published rubric (VOUCHED_CHECKLIST.md in the hub). The badge is awarded by maintainer review and attests to the specific version of the provider (recorded as vouchedVersion in the registry); later releases require a fresh request to maintain the endorsement. The rubric covers six items: multi-version conformance testing, documentation with use cases and limitations, security sign-off, Apache-2.0 licensing with DCO, MinEngineVersion declaration, and CsxFragment composition review. The badge is point-in-time: it vouches for the reviewed version only, not for future releases. Providers remain listed without the badge; the badge is optional, awarded post-listing, and signals that the maintainer has vetted the version's implementation quality against the published rubric.

Security review is non-negotiable for credential handlers

A provider that opens a database connection, joins a broker, or calls an authenticated service necessarily holds credentials at some point. The Vouched badge therefore requires an explicit security review covering credential lifetime, log redaction, and TLS defaults. Unbadged community providers are not gated on this, but the documentation makes the trust model clear to users who install them.

The Provider SDK and CONTRIBUTING guide. The frozen v1 provider contract is the Vouchfx.Sdk package (Apache-2.0; publication to NuGet.org is planned to accompany the engine's pre-release stream). Detailed authoring rules, the integration-test fixture pattern, and the Vouched-badge rubric are documented in the repository's CONTRIBUTING.md. The worked-example provider examples/Example.Steps.Echo demonstrates all four mandatory interfaces and documents the contributor's friction log in its README; Example.Steps.Hello is an even more minimal template. The integration-test fixture is a copyable template for every community provider.

The Provider Test Harness. Out-of-repo provider authors can unit-test a provider end-to-end without Docker by referencing the Vouchfx.Sdk.Testing NuGet package alongside Vouchfx.Sdk. This package exports ProviderTestHarness.RunSingleStepAsync(yaml, providerAssembly, stepId), which reproduces the engine's compile-and-run path for a single dependency-free step: schema-validate → parse → bind → validate → emit → compile-once → run-isolated → read outcome. Expected failures (schema or model validation) return as data with a null verdict and an error list; genuine Roslyn compile errors throw. The package also exports public stub implementations of the three frozen Vouchfx.Sdk context interfaces (TestBindingContext, TestProjectContext, TestCompileContext) in the Vouchfx.Sdk.Testing.Contexts namespace, removing the need for hand-written stub boilerplate in provider projects. Vouchfx.Sdk.Testing depends on three engine assemblies — Vouchfx.Engine.Abstractions, Vouchfx.Engine.Authoring, and Vouchfx.Engine.Compilation — published as a testing surface. This surface is versioned (semver, engine cadence) but not frozen, unlike the v1 Vouchfx.Sdk contract; a golden-file freeze gate over the test harness's own public signature enforces no breaking changes across v1.x (tests/Vouchfx.Sdk.Testing.Tests/SdkTestingContractFreezeTests.cs, golden file vouchfx-sdk-testing-public-api.v1.txt), scoped to the harness's own surface without freezing the evolving engine assemblies. The gate shares the SdkPublicApiSignature canonicaliser with the Vouchfx.Sdk freeze gate.

13.10 A worked example: the db-assert.postgres provider

To make the model concrete, this subsection walks through a single provider end to end — from the YAML an author writes, through the C# the provider implements, to the CSX the engine emits. It is deliberately abbreviated for illustration; the full source for each Core provider lives in the platform repository.

13.10.1 What the author writes

- id: assert-projection
  type: db-assert.postgres
  target: orders-db
  verifyMode: RETRY
  timeout: 20s
  query: |
    SELECT plan, billing_account_id
    FROM users
    WHERE id = @userId
  parameters:
    userId: "{newUserId}"
  expect:
    rowCount: 1
    row:
      plan: standard
      billing_account_id: "{billingAccountId}"

13.10.2 The typed model

public sealed record DbAssertPostgresModel(
    string Target,
    string Query,
    IReadOnlyDictionary<string, string>? Parameters,
    PostgresExpectation Expect,
    VerifyMode VerifyMode,
    TimeSpan?  Timeout)
  : IStepModel;

public sealed record PostgresExpectation(
    int? RowCount,
    IReadOnlyDictionary<string, object>? Row);

13.10.3 The schema fragment

{
  "$id": "https://schema.example.com/v1/providers/db-assert.postgres.json",
  "type": "object",
  "required": ["type", "target", "query", "expect"],
  "properties": {
    "type":       { "const": "db-assert.postgres" },
    "target":     { "type": "string" },
    "query":      { "type": "string" },
    "parameters": { "type": "object", "additionalProperties": { "type": "string" } },
    "expect": {
      "type": "object",
      "properties": {
        "rowCount": { "type": "integer", "minimum": 0 },
        "row":      { "type": "object" }
      }
    }
  }
}

13.10.4 Validation, resource declaration, and emission

The validator confirms that target names a dependency declared as a Postgres instance in the environment, that every parameter referenced in the query string appears in the parameters block, and that placeholders inside string values reference variables captured by earlier steps. The resource contributor declares that the provider requires a Postgres connection string for the named target. The compiler emits CSX that opens an NpgsqlConnection, executes the query with the parameter dictionary, and asserts the rowcount and field equalities; when verifyMode: RETRY is set, the whole block is wrapped in the Polly resilience policy that section 5 already provides. Two specific patterns in the emitted code are worth pointing out, because they differ from what a hand-rolled Aspire+Npgsql setup would write. First, the connection string is obtained through ((IResourceWithConnectionString)resource).GetConnectionStringAsync(ct) rather than a direct call on the concrete PostgresDatabaseResource: GetConnectionStringAsync is implemented explicitly on the interface in current Aspire releases and is not visible on the concrete type without the cast. Second, the disposable resources are not declared with using var because that form is not legal inside a Roslyn script body (see Section 13.3.1); the emitter uses plain var with explicit .Dispose() in a finally block.

// Illustrative emitted CSX (simplified). The step id 'assert-projection'
// is sanitised to 'assert_projection' before splicing into variable names.
{
    var conn_assert_projection = new NpgsqlConnection(
        Vars.ConnString("orders-db"));
    var cmd_assert_projection  = new NpgsqlCommand(@"SELECT plan, billing_account_id
                                                     FROM users WHERE id = @userId",
                                                  conn_assert_projection);
    try
    {
        await conn_assert_projection.OpenAsync();
        cmd_assert_projection.Parameters.AddWithValue(
            "userId", Vars.GetString("newUserId"));
        var row = await cmd_assert_projection.ExecuteSingleRowAsync();
        Assert.Equal("standard",                       row.GetString("plan"));
        Assert.Equal(Vars.GetString("billingAccountId"),
                                                       row.GetString("billing_account_id"));
    }
    finally
    {
        cmd_assert_projection.Dispose();
        conn_assert_projection.Dispose();
    }
}

That is the entire provider. The next provider in line — db-assert.sqlserver, db-assert.oracle, mq-publish.rabbitmq — is the same shape with the technology-specific parts replaced. The architecture has therefore turned the open-ended question “how do we support every database and every broker in the .NET ecosystem?” into a concrete, finite question that a single engineer can answer for one technology in a self-contained pull request.

14. Result Reporting and the Verdict Lifecycle

The platform's value is realised only when its outcome reaches the engineer who must act on it. Every section so far describes machinery that exists to produce a single thing: a trustworthy answer to the question “did the system under test do what we expected?” The reporting layer is the surface through which that answer becomes visible — and the surface through which the rest of the architecture either earns trust or loses it. A platform that orchestrates correctly, compiles deterministically, and asserts faithfully but then surfaces a failure as “step 4 failed: expectation not met” has wasted the work upstream, because the developer's next move is to bypass it and dig through raw logs. Reporting is therefore not a closing afterthought; it is a first-class architectural concern, designed with the same rigour as the compiler and the orchestrator.

This section defines the architecture's answer in five parts. It names the audiences the report serves, fixes the verdict taxonomy that classifies every outcome, defines the five-layer report architecture that carries data outward from a single step to a multi-week trend, specifies the structured event stream that every renderer and the Healer agent consume in common, and describes the rendering moves that turn the platform's deepest difficulties — asynchrony and cross-step state — into visible developer stories rather than forensic exercises. The section closes with the format spectrum mapped onto the permanently free OSS core and the planned Team and Enterprise tiers.

14.1 Reporting as a first-class architectural concern

Five audiences consume the report, and each takes something different from the same underlying data. The architecture serves all five from a single event stream, rendered differently for each consumer, because a separate report pipeline per audience would produce drift between what developers see and what dashboards count.

Audience	What they need from the report	Format that serves them
Developer debugging a failure	Immediate, in-context detail: what the step did, what it observed, what it expected, and the state of the world around it.	Terminal renderer; HTML report; VSCode Test Explorer inline.
CI system gating a build	Machine-readable verdict with deterministic exit codes; no prose.	JUnit XML; structured exit codes; JSON Lines event stream.
Team lead or QA manager	Aggregation across runs: flakiness, ownership, time-to-green, regression patterns.	Cross-run dashboard; trend charts; ownership reports.
On-call engineer triaging	A navigable artifact accessible from a different machine, with links into telemetry.	Hosted HTML report; observability deep links.
Auditor in a regulated industry	An attestable record of what was tested, when, against which version, with what outcome.	Immutable signed report; reproducibility envelope; SIEM export.

14.2 How the report applies the verdict taxonomy

The four verdicts are defined authoritatively in Section 12.1 — Pass, Fail, Environment error, and Inconclusive — and this subsection specifies what the reporting layer does with each of them rather than redefining them. The reporting layer's contribution to the taxonomy is presentational: distinct visual treatment, distinct counters, and distinct expansion defaults so that the four cases stay legible at a glance even when a run contains a mix of them.

Verdict	Visual treatment	Default rendering behaviour
Pass	Green check.	Collapsed by default; expandable for drill-down into captured variables and timing.
Fail	Red cross.	Expanded by default; carries the expected-versus-observed diff, the polling timeline if RETRY was in play, and the trace correlation id.
Environment error	Amber warning.	Counted separately from failures and listed in a dedicated section of the summary; carries the orchestrator's diagnostic and a link to the relevant container, broker, or tunnel log.
Inconclusive	Grey circle.	Counted separately again; carries the sub-case (timeout-without-response, partition-outlasted-grace, or upstream-capture-unmet) and, for sub-case (c), the id of the earlier step whose capture was unmet.

Separating defect from environment failure is the most important contribution the reporting layer makes here, and it is the contribution most existing testing tools fail at. A failure that says “your Kafka container did not start” is fundamentally different from “your code is broken,” and a report that conflates the two destroys trust in the tool within days. Inconclusive deserves the same care: a step skipped because an earlier capture was unmet is not a defect in the system under test, and rendering it as a failure would falsely accuse a downstream component of misbehaviour. The conservative behaviour — four verdicts, four colours, four counters — is what keeps each case legible. In the planned cloud-backed Team and Enterprise tiers, these classifications would also drive notification routing into the customer's existing alerting channels; in the free local core they are purely presentational.

14.2a Terminal renderer accessibility (WCAG 1.4.1)

The terminal renderer output faces a special accessibility requirement: verdicts must never be distinguished by colour alone. The implementation carries each verdict as a distinct text token (PASS, FAIL, ENV_ERROR, INCONCLUSIVE) unconditionally in both plain and decorated output — a WCAG 1.4.1 guarantee that survives colour-blindness, screen readers, and terminal limitations. The mapping is:

Verdict	Text token	ASCII glyph (decorated)	ANSI colour (decorated)
Pass	PASS	[+]	Green (SGR 32)
Fail	FAIL	[x]	Red (SGR 31)
Environment error	ENV_ERROR	[!]	Yellow (SGR 33)
Inconclusive	INCONCLUSIVE	[?]	Blue (SGR 34)

Decorations (glyph + colour) are enabled only when the following conditions all hold: the output is an interactive terminal (Console.IsOutputRedirected is false), the NO_COLOR environment variable is unset, and the --no-decorations CLI flag is not passed. Piped, redirected, CI, and test output therefore defaults to plain text (text tokens only, no colour or glyphs). The CLI's decoration logic resides in the runner layer — the renderer itself is a pure function that obeys the decorate boolean parameter — so the rendering remains deterministic and unit-testable. The glyph is a shape cue independent of colour; even a colour-blind user with decorations enabled sees a distinct shape + token per verdict. The colour is a redundant, sighted-only convenience layered on top of the always-present text token, ensuring that no information is conveyed through colour alone.

14.3 The five-layer report architecture

Reports are best thought of as five concentric layers, each layer an aggregation or derivation of the one inside it. Designing them as concentric rather than as five separate artifacts is what keeps the system coherent: the same data that drives a developer's terminal also feeds the team lead's dashboard, and the live execution feed is not a separate code path but the inner layers streamed as they are produced.

Layer	What it contains	Primary consumer
Per-step record	Structured object capturing what the step tried, what it observed, what it expected, the verdict, the duration, every captured variable, every RETRY attempt with its timestamp and observation, and the correlation ids that link to engine telemetry.	The renderers; the Healer agent.
Per-scenario report	One YAML file's execution rendered in full: all step records in order, the cross-step state thread woven through, the environment summary, and the reproducibility envelope.	Developer debugging a failure.
Suite-level summary	Aggregation across the scenarios in one run: headline counts by verdict, the failing files, total duration, machine-readable exit code.	CI system gating the build.
Cross-run history	Trend data across many runs: per-test flakiness scores, time-to-green after a fix, ownership distribution, regression patterns, drift detection.	Team lead or QA manager.
Live execution feed	The inner layers streamed as they are produced: what the author sees in the terminal or VSCode during a long-running suite.	Author watching the suite run.

The live execution feed earns special mention because long suites without it become opaque, and opacity drives authors to add Console.WriteLine statements to their tests — the exact behaviour the platform exists to make unnecessary. The feed is therefore not optional; it is the inner layers of the report rendered in real time, with the same data the offline report would later show.

14.4 The structured event stream as substrate

Every renderer — the terminal, the HTML report, the JUnit XML, the cloud dashboard, the Healer agent — consumes the same underlying event stream. The stream is JSON Lines, one event per line, schema-versioned. This single-substrate decision is the architectural commitment that prevents reporting drift: a new renderer can be added without changing the engine, a new event type can be added without breaking existing renderers (because the schema is versioned and renderers tolerate unknown fields), and the same data that drives a human-readable terminal also drives a machine-consumable agent.

// Example events from the structured stream (illustrative)
{"v":1, "type":"suite-started",    "runId":"r-7f3a", "engineVersion":"1.0.0", "schemaVersion":"v1",
                                    "providers":[{"kind":"http.rest","version":"1.0.0"}, ...] }
{"v":1, "type":"scenario-started", "scenarioId":"users-e2e",  "file":"tests/users.yaml",
                                    "contentHash":"sha256:9e1c..." }
{"v":1, "type":"step-started",     "stepId":"expect-billing-event", "kind":"mq-expect.kafka",
                                    "verifyMode":"RETRY", "timeoutMs":30000 }
{"v":1, "type":"step-attempt",     "stepId":"expect-billing-event", "attempt":1, "tMs":540,
                                    "observation":{"matched":0} }
{"v":1, "type":"step-attempt",     "stepId":"expect-billing-event", "attempt":2, "tMs":1530,
                                    "observation":{"matched":1, "diff":[{"path":"payload.status",
                                                                            "expected":"PENDING",
                                                                            "observed":"PROCESSING"}]} }
{"v":1, "type":"step-completed",   "stepId":"expect-billing-event", "verdict":"FAIL",
                                    "durationMs":30024, "observation":{"matched":1,"diff":[...]},
                                    "correlationIds":{"trace":"00-9e1c..."} }
{"v":1, "type":"reproducibility-envelope", "runId":"r-7f3a", "scenarioId":"users-e2e",
                                    "envSchemaVersion":"v1", "secretReferences":[{"source":"env",
                                    "referenceHash":"sha256:a3f2..."}], "fixtures":[{"reference":
                                    "fixtures/users.sql", "contentHash":"sha256:d4c8..."}] }
{"v":1, "type":"scenario-completed", "scenarioId":"users-e2e", "verdict":"FAIL",
                                    "counts":{"pass":2,"fail":1,"envError":0,"inconclusive":0} }

The reproducibility-envelope event carries per-scenario reproducibility metadata: the hashed secret references (never resolved values) and the content hashes of applied fixtures (see Section 14.7 and Section 17), enabling readers to reproduce the exact run on a different machine.

Two design decisions in this stream are worth underlining. First, every step-attempt event is recorded individually rather than collapsed into a summary: this is what makes the polling timeline of the next subsection possible without re-running the suite. Second, every event carries a correlation id that resolves to a trace in the observability stack of Section 12, so the report becomes a navigable index into the rest of the platform's data plane rather than a dead end.

An optional third point: step-completed events may also carry a observation field — the structured provider observation (e.g. a failed assertion's expected-vs-observed diff) — which renderers use at render time to compute human-readable diffs (Section 14.10). This field is omitted when the step recorded no observation; it is purely structured data, never rendered diff text. Adding this optional field preserves backward compatibility — older renderers ignore it via [JsonExtensionData] — and costs the engine only the work of forwarding the observation from the runtime to the event.

Persistence to disk: The raw event stream can be persisted to a file via the CLI's --events <path> option (alias --json), which writes the buffered stream verbatim — one JSON object per line, UTF-8 without a BOM — for consumption by downstream tooling such as the VSCode Test Explorer. This is a purely additive facility that re-emits the frozen v1 stream byte-for-byte without any record or wire-contract change; the stream itself remains a private implementation detail rendered differently by each consumer, and persisting it to disk does not alter that contract. Security note: Unlike HTML and JUnit reports (which summarise step observations to their structure, never their values), the event stream persists step observations verbatim; authors using script.csharp steps must not embed revealed secret values in thrown exception messages, since those messages become step observations in the stream.

14.4.1 Environment-error events and image-pull diagnostics

The event stream includes an environment-error event type, emitted when orchestration infrastructure fails and prevents a scenario from running. The event carries an errorKind (e.g. ImagePull, HealthGate, Discovery, Provision) and an optional authStatus field that further refines image-pull failures. The authStatus field distinguishes five outcomes:

authStatus value	Meaning
"unauthenticated"	The registry rejected credentials with HTTP 401 or equivalent.
"access-denied"	The registry rejected access with HTTP 403 or equivalent.
"rate-limited"	The registry rejected the pull due to rate limiting (HTTP 429 or toomanyrequests message); the canonical Docker Hub cold-runner failure (MVP §10 named risk).
"anonymous"	The pull was attempted without credentials and failed for a non-authentication reason (image not found, manifest missing, etc.).
null	The registry host was unreachable via DNS or network (connection refused, no route to host, timeout, etc.); no authentication was attempted. Also null for non-ImagePull kinds (HealthGate, Discovery, Provision). Omitted from the wire when null.

The authStatus field is omitted from the JSON-Lines wire when null. This granular classification lets developers distinguish between credentials that need rotation ("unauthenticated", "access-denied"), infrastructure that needs scaling ("rate-limited"), and connectivity problems (null) — avoiding trial-and-error fixes.

14.4.2 The v1 event-wire contract freeze and the scenarioId decision

The event-record surface — the top-level event records and all nested value records (EventEnvelope, ScenarioStartedEvent, StepStartedEvent, StepAttemptEvent, StepCompletedEvent, ScenarioCompletedEvent, ReproducibilityEnvelopeEvent, VerdictCounts, CapturedVar, SubstitutionRef, SecretReferenceDigest, FixtureDigest) — is frozen at v1 for the v1.x engine series, enforced by a golden-file CI gate (EventContractFreezeTests) that asserts every property name, CLR type, and [JsonPropertyName] wire key is byte-for-byte identical to the committed golden. Any change to the JSON-Lines wire contract breaks every consumer (terminal renderer, HTML report, JUnit XML, dashboard, Healer agent), so this freeze is non-negotiable.

A deliberate architectural decision is encoded in the golden and pinned independently: the three step events — StepStartedEvent, StepAttemptEvent, StepCompletedEvent — carry RunId and StepId but do not carry ScenarioId. The terminal renderer's (runId, stepId) cache already disambiguates an aggregated or parallel multi-scenario stream, because each scenario has a distinct runId. Re-adding ScenarioId to a step event would be a conscious, reviewed change (the golden freeze gate would fail, and the wire contract would increment), not an accidental drift. Designing the event stream to omit this redundant field leaves room for future optimisations and keeps the event payload lean.

14.5 Rendering asynchronous flows: the polling timeline

When a RETRY step fails, the developer needs to see what happened, not just that it failed. Most existing integration testing tools report only the final pass-or-fail at the end of a timeout window, leaving the developer to read raw broker logs to understand why. The reporting layer here renders the full polling timeline directly: each attempt, its timestamp relative to step start, the observation made on that attempt, and how the observation differed from the expectation. Because the engine's Polly-backed RETRY machinery has already recorded every attempt in the structured stream, the polling timeline costs the architecture nothing to produce; the commitment is to surface it prominently.

Step  expect-billing-event  (mq-expect.kafka)        FAIL  (timeout after 30.0s)

      t=  0.5s   attempt 1   no message matched the key {newUserId}
      t=  1.5s   attempt 2   1 message matched, payload.status was "PROCESSING"
                             expected "PENDING"
      t=  4.0s   attempt 3   no message matched
      t= 10.0s   attempt 4   no message matched
      t= 20.0s   attempt 5   no message matched
      t= 30.0s   attempt 6   timeout fired

This rendering is the single most distinguishing feature of the report against the existing testing tool market. It turns asynchrony from a forensic exercise back into a story the test author wrote, with no agentic component required: every attempt is recorded by the engine's Polly-backed RETRY machinery as a structured event, and the renderer reads them directly. The MVP ships this rendering in full. As a sibling feature, the captured-variable provenance thread (§14.6) shows where each value in the scenario originated and was threaded through steps, making cross-step state visible in the same spirit.

In later tiers, once the agentic Healer is available, the same event stream supports a second class of content layered on top: a hypothesis line in which the agent proposes a likely cause from the attempt pattern. The example below shows what a future-tier rendering might look like; nothing in the timeline above changes, and the hypothesis is clearly marked as agent-generated and reviewable.

      [agent hypothesis]  the system produced one matching event briefly, then
                          stopped — a transient state that vanished before the
                          timeout. This is the signature of a retry-and-overwrite
                          in the upstream service rather than a missing event.

14.6 Cross-step state visibility: the captured-variable thread

A test that fails on step 4 is usually failing because of something that happened in step 2. The reporting layer makes that explicit: for each step it shows what variables were captured (with their values), and for each step it shows what variables were substituted into it (with their sources). The result is a thread of state through the scenario, written in the same vocabulary the author wrote in, collected by the same capture-and-substitute machinery defined in the DSL specification. A reader can trace the lineage of any value backward to the step that produced it.

The terminal renderer emits a provenance: section under each step that shows this lineage. For each captured variable it renders captured '<name>' <- step '<id>' (<path>) with an optional (no match) marker if the JSONPath or XPath matched nothing. For each substitution it renders either substituted '{placeholder}' (from step '<id>') -> step '<current>' for ordinary placeholders or substituted ${secret:<reference>} (redacted) -> step '<current>' for secret-derived substitutions, never revealing the secret's value. The section is omitted when a step has no captures or substitutions, keeping straightforward runs uncluttered.

Scenario  users-e2e

  1. create-user             PASS (203ms)
    provenance:
      captured 'newUserId' <- step 'create-user' ($.id)

  2. expect-billing-event    FAIL (30024ms)
    provenance:
      substituted '{newUserId}' (from step 'create-user') -> step 'expect-billing-event'
      captured 'billingAccountId' <- step 'expect-billing-event' ($.account.id)   (no match)

  3. assert-projection       INCONCLUSIVE (skipped)
       reason  depends on billingAccountId from step 2, which was not captured

The thread above does more than show variables; it shows why step 3 is inconclusive rather than failed. A scenario whose later steps depend on values that earlier steps did not produce is not a scenario whose later steps are broken. Rendering this distinction prevents a single root-cause failure from cascading into a wall of red unrelated to the real defect. Secret-derived substitutions are always rendered redacted (reference label only, never the value), preserving the security invariant even in the report (§17).

14.7 The reproducibility envelope

Every report carries enough context that a reader can reproduce the exact run on a different machine. The envelope contains the engine version, the schema version, the StepKindRegistry's frozen list of provider names and versions, the container image digests resolved by the orchestrator at run time, the seeded variables and their values, the content hash of every YAML file in the suite, and the identities of any customer DLLs referenced through dynamic assembly resolution. The envelope turns a failure report from “this happened once” into “this can happen again, here is exactly how.” For regulated industries it is a strong reproducibility artifact, though not a formally attestable one: regulated audits typically require a third-party timestamping authority or a tamper-evident audit log outside the platform's control, and the platform does not yet provide either. Customers needing those additional guarantees layer them on top of the envelope. For non-regulated teams the envelope eliminates the “works on my machine” debate before it begins.

14.8 Linkage to engine telemetry and to the Healer agent

The report does not contain raw logs; it contains links to them. Engine telemetry — traces, container logs, broker state at the moment of failure, load injection profiles from the Performance LAB — lives in the observability stack of Section 12. The report surfaces correlation ids at each level: per-step records carry trace ids that resolve to spans, per-scenario reports link to the orchestrator's container log bundle, suite summaries link to the load profile if performance testing ran. The report is therefore a navigable index into the rest of the platform's data plane, not a duplicate of it.

The Healer agent of Section 8 consumes the same structured event stream the developer's report renders from. When the agent receives a failure event with a structured diff between expected and observed, it can hypothesise a fix and post it as a healer-suggestion event back into the stream; when it receives only a “fail: assertion did not hold,” it cannot. Making the event stream rich enough to support both human reading and machine reasoning is therefore a single architectural commitment, not two. The renderers display healer suggestions inline with the failure that prompted them, clearly marked as machine-generated and reviewable.

14.9 Format spectrum and tier mapping

The five-layer model maps naturally onto the platform's architecture, where the inner layers serve the developer and form the permanently free Apache-2.0 open-source core, and the outer layers enable team and organisational capabilities planned for future commercial offerings.

Scope	Reporting capabilities included
Permanently free OSS core	Rich terminal renderer with colour and the polling timeline; HTML report written to disk; JUnit XML for CI gates; VSCode Test Explorer integration with inline decorations on failing YAML lines; the live execution feed in both the terminal and the editor.
Planned Team tier	Everything above, plus a hosted cross-run dashboard: flakiness scores, time-to-green, ownership distribution, regression detection, and trend charts. Reports forwarded to the cloud backend on every run with no change to author workflow.
Planned Enterprise tier	Everything above, plus RBAC on report access, audit-grade immutability of historical records, structured SIEM export (Splunk, Datadog, native), retention policies aligned to compliance frameworks, and signed reports for attestation. The reporting layer joins the rest of the organisation's data plane rather than living as an island.

14.10 Provider participation in reports

Providers contribute to the report through their typed model and through an optional renderer hook. The structured-diff machinery for db-assert.postgres knows how to render “expected row count 1, observed 0” or “row[0].plan: expected 'standard', got 'trial'” in a way that respects relational semantics; db-assert.mongodb renders a document diff with array indexes and nested paths; db-assert.redis renders a key-and-value diff. A provider's implementation can supply an IStepDiffRenderer interface (Section 13.8.1, an additive optional interface) implementing bool CanRender(JsonElement observation) and string? RenderDiff(JsonElement observation). The renderer calls CanRender to determine whether it can render the step's structured observation, and if true, calls RenderDiff to produce the rendered diff text. This interface is discovered at startup from the provider's type like the core compilation interfaces, but runs in the engine's Default AssemblyLoadContext so it raises no memory-model concern (Section 5.4). The structured observation is a plain System.Text.Json.JsonElement rather than a provider-specific type, keeping the reporting layer decoupled from Vouchfx.Sdk; the engine invokes the renderer through a delegate seam built at startup from the frozen StepKindRegistry. RenderDiff returns LF-separated lines (no trailing CR) ready to splice under the step line in the terminal output. This keeps reporting fidelity high without putting database-specific knowledge in the report renderer itself, and it means a community contributor adding a new provider also contributes the right rendering for its failure mode — not as an afterthought, but as part of what the provider contract asks for.

14.11 Accessibility commitments for the reporting surface

Because the report is a user surface that pilots and customers will read every day, the architecture commits it to WCAG 2.1 AA conformance at v1.0. Concretely: the four verdicts are conveyed through both colour and shape (and through plain-text labels in monochrome renderings) so that the distinction survives colour-blindness and minimal terminals; the HTML report is keyboard-navigable end to end with a logical tab order and semantic markup (headings, lists, tables) rather than div-soup; and the terminal renderer offers a screen-reader-friendly mode that omits decorative box-drawing characters and renders the polling timeline as a numbered list. These commitments cost little if designed in from the start and are expensive to retrofit later; they are stated as architecture-level commitments here rather than as a post-launch task. For the complete WCAG 2.1 AA conformance record, including audit method, remediation history, and CI gates, see docs/accessibility.md.

The report is the window onto the experiment

Every other section of this document describes how the platform runs the experiment correctly. This section describes how the platform tells the experimenter what happened. The architectural commitment is that those two concerns are equal: a brilliant test platform that produces unreadable reports is not a useful one. Reporting is therefore designed in, not bolted on.

15. Test Data, Seeding, and Test Doubles

Every section so far has assumed that the data a test queries, publishes, or asserts against simply exists. In practice, getting realistic data into a distributed system is often half the work of testing it, and an architecture that leaves the question unanswered forces every team to invent its own seeding pattern — with the predictable result that the patterns are ad hoc, mutually incompatible, and eventually blamed on the platform. This section defines how state is established before a test runs, and the platform's deliberate position on test doubles for dependencies that cannot or should not be exercised for real.

15.1 The seeding problem

A scenario that asserts “expect a billing event for user u-8af2-c13e” presupposes that the user exists, that reference data such as currencies and product catalogues is loaded, and that any upstream state the system depends on has been established. Respawn, described in the orchestration layer, resets databases to a pristine state between tests; nothing described so far establishes state before them. Seeding is the missing half of that lifecycle, and the architecture treats it as a first-class part of environment preparation rather than as something each test improvises.

15.2 Four seeding strategies, one default

The platform recognises four established strategies and is opinionated about which is the default, while keeping the others available as escape hatches for cases the default does not fit.

Strategy	How it works	When it is the right choice
Declarative fixtures (default)	An environment-level seed block declares reference SQL files, JSON or BSON document fixtures, and warm-up messages to publish to brokers before the suite starts. The engine applies them after the topology is healthy and before the first step runs.	Reference and lookup data; a known baseline shared across a suite; anything that is the same for every run.
Programmatic factories	A script step early in the scenario constructs entities through the system's own APIs — the same path a real client would take — capturing the generated identifiers into the variable context for later steps.	Entities whose creation should itself be exercised; data that must be realistic rather than hand-written; per-scenario unique state.
Snapshot and restore	The orchestrator provisions a dependency from a pre-baked image or volume snapshot that already contains a representative dataset, rather than seeding an empty store.	Large datasets too slow to load row by row; data shaped like production for realistic query behaviour.
Anonymised production extract	A snapshot derived from production data with PII removed by an offline pipeline, loaded as in the snapshot strategy.	Realism that synthetic data cannot reproduce; enterprise teams with a sanctioned anonymisation pipeline. Out of scope for the MVP; named so the door is not closed.

The declarative fixture is the default because it is the most reproducible: it lives in source control next to the test, it is visible in code review, and it produces the same baseline on every machine. The seed block's precise grammar is defined in the DSL specification; here the relevant architectural point is that seeding runs inside the same health-gated lifecycle as the rest of orchestration, so a seed that fails to apply produces an environment error — not a confusing assertion failure three steps later.

15.3 Seeding and the verdict taxonomy

A failure to seed is an environment error, never a test failure. If a reference SQL file cannot be applied because the database did not accept it, the scenario never truly ran, and the report must say so in the same vocabulary it uses for a container that failed to start. This keeps the seeding mechanism honest: it cannot silently corrupt a run into a false pass or a misattributed fail. The reproducibility envelope of Section 14 records the content hash of every applied fixture, so a reader of a report knows exactly what baseline the run started from.

15.4 The platform's position on test doubles

The deepest design commitment of this platform is to test against real dependencies; that is the entire purpose of the orchestration layer. But three situations make a real dependency impractical: a downstream system that costs real money to call (a payment gateway), one that is undesirable to hit (a third-party API with rate limits or no sandbox), and one that does not yet exist (contract-first development against a planned service). The platform needs a stated position on these, because silence produces inconsistent practice.

The stance: doubles are containers, not a platform feature

The platform ships no built-in mocking framework. A test double is provisioned the same way any other dependency is — as a Testcontainer running WireMock, Mountebank, or an equivalent — declared in the environment section like a real dependency and addressed through the same service discovery. For in-process stubbing of a customer's own types, the script step is the escape hatch. This keeps the platform's core honest about what it is: an integration tester against real infrastructure, with doubles as a deliberate, visible exception the author opts into rather than a default the platform encourages.

This position has a useful consequence for the provider model. Because a double is just a container speaking a real protocol, an http.rest step does not know or care whether it is talking to a real service or a WireMock stub — the test is written identically, and swapping the double for the real service later requires no change to the step. The seam is in the environment declaration, not in the test logic, which is exactly where a seam belongs.

16. The Test Runner: Selection, Parallelism, and Isolation

The command-line runner was described earlier as a headless executor with deterministic exit codes. That is true but understates it: a test runner is a product in its own right, and at the scale a successful adoption reaches — hundreds of test files within eighteen months — the runner's selection model, parallelism, and isolation guarantees determine whether the platform remains usable or becomes something teams route around. This section specifies the runner as the first-class surface it is.

16.1 Test selection

A team with five hundred test files never wants to run all of them on every change. The runner therefore exposes a selection language that composes several independent axes, so that a developer, a CI pipeline, and a nightly job can each ask for exactly the subset they need. The first four axes operate on metadata already in the file or the source tree and are available in the permanently free core. The fifth depends on persisted run history; local prior-verdict selection is in the core, whilst the full history form is planned for future commercial offerings.

Selection axis	Example intent	How it is expressed	Availability
By tag	“Run the smoke tests only.”	Tags declared in each file's metadata block; the runner takes an include and exclude tag expression.	Free core
By ownership	“Run the payments team's suites.”	An owner field in metadata; the runner filters on it.	Free core
By path	“Run everything under tests/billing.”	Glob patterns over the file tree.	Free core
By change	“Run only what this branch touched.”	The runner diffs against a base ref and selects files changed plus, optionally, files whose declared dependencies changed.	Free core
By prior verdict (last run)	“Run the suites that failed last time.”	The core writes the most recent run's structured event stream to a local cache; the runner reads it and selects by prior verdict. Only the immediately previous run is retained locally.	Free core
By prior verdict (across history)	“Run the suites that failed in any of the last seven days.”	Backed by the hosted run-history store; selection is over the full retained window.	Planned Team tier

The axes compose: a CI job can ask for “tagged regression, owned by payments, changed in this branch” in a single invocation. Selection operates on metadata the author already writes, so it costs the author nothing beyond tagging discipline.

16.2 Parallelism and the concurrency budget

Scenario parallelism is opt-in through the --parallel <n> CLI flag (§13.5); when omitted, scenarios run sequentially against one shared topology. Independent scenarios can run concurrently when the flag is supplied, bounded by the supplied concurrency degree. Parallelism is across scenarios, not within a scenario: the steps of one scenario remain a strict ordered sequence, because their state dependencies require it. Each concurrent scenario builds, owns, and disposes its own isolated Aspire topology — no shared-state reset or database reset between scenarios is required because each scenario starts with a clean topology and its fresh dependencies. The default concurrency when --parallel is supplied is Math.Max(1, Math.Min(ProcessorCount, 4)) — a conservative cap because containers, not CPU, are the scarce resource: a topology of four to six containers multiplied by an unbounded fan-out would exhaust a typical developer laptop's memory and Docker resource limits within seconds. The deterministic single-render model (events concatenated in declaration order) and complete-all cancellation semantics (every topology disposes, never fail-fast) are preserved across all concurrency levels. Timeout or cancellation of one scenario never cancels siblings; an escaping engine exception is mapped to EnvironmentError rather than Fail (§12.1), so parallelism never manufactures defects from infrastructure faults.

A note on the event stream schema: step events do not carry a scenarioId field. Each scenario receives a distinct runId, so the reporting layer's (runId, stepId) cache key already disambiguates steps across an aggregated multi-scenario stream. The v1.x schema is frozen (§8.3); a future schema-evolution decision record will formalise this for v2.

16.3 Isolation between scenarios

The isolation contract must be stated explicitly, because it is the assumption every test author reasons about and a silent default is how flakiness gets seeded into the platform itself. The contract has three parts.

First, each scenario gets a clean data baseline. Respawn-based reset runs between scenarios on relational dependencies; for stores Respawn does not cover, the runner uses a per-store-type cleanup strategy described in the orchestration appendix. Second, scenarios may share a topology or be given fresh ones, and this is a declared choice: in sequential runs, by default a suite shares one provisioned topology across its scenarios for speed, with per-scenario data reset providing the isolation; a scenario that needs genuine infrastructure isolation — a destructive test, or one that reconfigures a broker — declares that it requires a private topology and the runner honours it. In parallel runs the picture changes because parallel scenarios cannot safely share a mutable topology. Third, parallel scenarios use database-namespace isolation by default and full topology isolation on request. The default for a parallel run is the cheaper option: scenarios share the topology but each scenario gets a private database schema (Postgres, SQL Server) or a private collection prefix (MongoDB) or a private Redis database number, so writes from one scenario are invisible to another without paying for a separate Postgres process. A scenario that genuinely requires a private topology — because it reconfigures a broker, or because the system under test maintains process-wide state across schemas — declares that explicitly and the runner provisions one for it. The expensive default of “private topology per parallel scenario”, which earlier drafts of this section described, is the explicit opt-in, not the default; the cheaper namespace-isolation default is what makes parallelism actually usable on developer hardware.

16.4 Exit codes

The runner's exit codes carry the verdict taxonomy all the way to the CI system, so that infrastructure breakage, inconclusive results, and genuine defects can be distinguished and handled separately. The mapping is controlled by two optional flags, both off by default, which implement the principle that only Fail breaks CI by default.

Exit code	Verdict	Meaning	Opt-in flag
0	Pass, EnvironmentError, Inconclusive	All scenarios passed, or only non-breaking verdicts occurred.	–
1	Fail	One or more scenarios failed — a genuine defect in the system under test. Breaks CI by default.	–
2	UsageError	Bad arguments, missing path, invalid --parallel value, or --watch combined with --parallel. The suite never ran.	–
3	EnvironmentError	The aggregate verdict was EnvironmentError (infrastructure breakage: unhealthy container, image-pull failure, seed failure, tunnel collapse). Off by default; breaks CI only when --fail-on-env-error is set.	--fail-on-env-error
4	Inconclusive	The aggregate verdict was Inconclusive (timeout, network partition outlasted grace, upstream capture unmet — the engine could not determine correctness). Off by default; breaks CI only when --fail-on-inconclusive is set.	--fail-on-inconclusive

The two opt-in flags are:

• --fail-on-env-error — When set, an EnvironmentError verdict exits with code 3 instead of 0. Use this to gate on infrastructure failure in environments where infrastructure reliability is the team's responsibility.
• --fail-on-inconclusive — When set, an Inconclusive verdict exits with code 4 instead of 0. Use this to gate on results the engine could not decide (timeout, partition, unmet upstream captures), in scenarios where deterministic decisions are required.

The distinct codes 3 and 4 sit deliberately above the UsageError code (2) so there is no collision: 0 = success, 1 = a product defect, 2 = a usage error, 3 = infrastructure broke, 4 = the engine could not decide. This separation lets a CI system act on each outcome independently: fail the build on Fail (1), notify on-call about EnvironmentError (3), or escalate Inconclusive (4) to a reliability engineering team. The principle is that a team should be able to see infrastructure problems for what they are, not buried under a wall of failing tests.

16.5 Deliberate non-features

Two capabilities are withheld on purpose, and naming them prevents their absence from looking like an oversight. The runner does not automatically retry a failed test, because automatic retry hides flakiness rather than surfacing it, and a platform whose entire premise is trustworthy verdicts must not bury its own non-determinism. The runner also does not reorder tests to optimise runtime in a way that could change outcomes; selection is explicit and ordering within a scenario is fixed. A watch mode for fast local iteration and a CI sharding interface for splitting a large suite across parallel CI agents are in scope as conveniences, but neither changes the verdict a test produces.

16.6 GitHub Actions integration

The vouchfx CLI ships with a reusable GitHub Actions workflow (.github/workflows/vouchfx-run.yml) that runs a .e2e.yaml suite end-to-end, publishing JUnit XML and self-contained HTML reports as artefacts. Consumers in other repositories can call this workflow with uses: to integrate vouchfx into their CI pipelines. The workflow parameterises the scenario directory path, the vouchfx source repository and reference, the .NET SDK version, the --fail-on-env-error and --fail-on-inconclusive gating flags, optional container image pre-warming (to mitigate Aspire/DCP's ~20 second per-resource cold-start watchdog), and the runner label; the JUnit XML and HTML report artefacts are produced to fixed paths (results.xml and report.html). The workflow builds vouchfx from source (deliberately, pinned-ref lock-step between engine and workflow, even though vouchfx is also distributed as a packaged NuGet global tool for local use) and respects the exit-code gating semantics (section 16.4): only Fail breaks CI by default; EnvironmentError (3) and Inconclusive (4) are opt-in via the respective flags. A reference caller is provided at .github/workflows/vouchfx-run-reference.yml, running this repository's own smoke suite to prove the workflow end-to-end. Supply-chain guidance in the README recommends pinning the uses: reference and each prewarm-images entry to immutable commit SHAs and image digests respectively, rather than floating tags or branches.

17. Secrets and Configuration Management

Tests need secrets: database connection strings, API keys for the system under test, OAuth client secrets, signing keys for verifying webhook authenticity. An architecture that does not give these a first-class home guarantees that teams will hard-code them into YAML, that the platform's security review will go badly, and that the first published incident will be a credential committed to a public repository. This section defines where secrets come from, how they are referenced without entering source control, and how they reach the steps that need them.

17.1 The reference syntax

Secrets are never written literally in a test file. A field that needs one carries a reference of the form ${secret:source/path}, which the engine resolves at runtime from a configured secret source immediately before the step that uses it executes. The resolved value lives only in memory for the duration of the run and is never written into a report, a log, or a captured-variable thread — the reporting layer redacts any value that originated from a secret reference, showing ${secret:…} in its place. The reference is what appears in source control; the value never is.

environment:
  services:
    orders-api:
      type: dotnet-service
      env:
        ApiKey: "${secret:vault/orders-api/key}"
  dependencies:
    orders-db:
      type: postgres
      connectionString: "${secret:env/ORDERS_DB_CONN}"

17.1.1 Resolution happens at step execution, not at compile time

An important architectural point: secrets are not interpolated into the generated CSX at compile time. If they were, the resolved values would be baked into the compiled assembly's IL — which would defeat the compile-once optimisation of Section 5 (every run with different secrets would produce different IL), corrupt the reproducibility envelope (its content hash would depend on values that should never appear in it), and force a recompilation on every secret rotation. Instead, the compiler emits a call to the global context's Secrets accessor, and resolution happens at the moment the step executes. The reproducibility envelope therefore hashes the secret references — the YAML strings — not the resolved values, which is exactly the correct behaviour.

// What the engine emits for a step that uses a secret reference (illustrative)
{
    var apiKey = Vars.Secrets.Resolve("vault/orders-api/key");
    using var client = Vars.Http("orders-api");
    client.DefaultRequestHeaders.Authorization =
        new AuthenticationHeaderValue("Bearer", apiKey);
    // ... rest of the step ...
}

The Vars.Secrets.Resolve call is intercepted by the reporting layer's redaction filter: if the resolved string appears in any subsequent observation, captured variable, log line, or assertion message produced by the step, it is replaced with ${secret:vault/orders-api/key} in the report. This guarantees redaction at the renderer level rather than relying on the provider to remember to redact — a discipline that, in practice, providers will sometimes forget.

17.1.2 Defence-in-depth scrubbing of provider observations

Type-based SecretString redaction is the primary mechanism: every structured surface (event fields, captured-variable thread, terminal/HTML/JUnit renderers) is redacted by construction because it only handles the SecretString carrier or non-sensitive metadata. The one surface the engine cannot type-check is free-form provider observation text — a provider builds observations by hand, so the engine cannot guarantee they contain only safe data. The script.csharp path is the most acutely at risk: when an author catches an exception and assigns __ex.Message to the observation, that message is spliced verbatim, potentially carrying a secret value that the exception handler did not redact.

A per-scenario ResolvedSecretLedger (owned by the SecretAccessor) records every value successfully revealed by Resolve, and the runner applies this ledger as a defence-in-depth scrub at the reporting boundary: before a step's observation text enters the JSON Lines event stream, every occurrence of a recorded secret value is replaced with ***REDACTED*** in both its raw form and its JSON-escaped form. This catches the realistic accident — a secret appearing unredacted in an exception message — without chasing theoretical transforms that are explicitly out of scope.

Importantly, the ledger deliberately does not recognise transforms of a revealed value: base64 encodings, HMAC signatures, substrings of a secret. Once an author invokes Reveal() and reshapes the bytes, the engine has no value to recognise, and the ledger cannot match it. Authors who deliberately Reveal() a secret and then transform it (for example, to compute an HMAC that proves possession) remain responsible for ensuring that transformation, log message, or stack trace does not leak the input. This is the documented, auditable escape hatch (the single call to Reveal() in a provider's source) and a developer responsibility, not an engine guarantee.

The memory model is clean: the ledger holds plain System.String values in the Default ALC (owned by the SecretAccessor, which the runner holds), so recording from inside the collectible script's accessor.Resolve(…) call runs a Default-ALC method body that adds to a Default-ALC set. No static handles bridge the collectible-ALC boundary, and the ledger is collected when the per-scenario accessor is released.

17.2 Pluggable secret sources

The part of the reference before the slash names the source, and sources are pluggable so that the same test file works in different environments by changing configuration rather than content. The MVP implements two sources, with two more reserved for future development.

Source prefix	Resolves from	Availability	Status
env	An environment variable on the machine running the suite. The simplest source; suitable for local development and basic CI.	Free core	MVP
vault	HashiCorp Vault KV v2, addressed by logical path and field selector. Configuration via three environment variables: VAULT_ADDR (server address, e.g. http://127.0.0.1:8200), VAULT_TOKEN (access token), and VAULT_KV_MOUNT (KV v2 mount name; defaults to secret). Resolution happens at step-execution time; the token and resolved value never leak into logs, reports, or captured variables. Reference form: ${secret:vault/<kvPath>#<field>} (e.g. ${secret:vault/secrets/api-keys#admin-token}).	Free core	MVP
file	A local secrets file outside the repository, git-ignored by convention. Convenient for a developer's standing local credentials.	Free core	Deferred
cloud	A cloud provider secret manager — Azure Key Vault, AWS Secrets Manager — selected by configuration.	Planned commercial tiers	Deferred

Because the source is chosen by configuration and only the logical path appears in the file, a test authored against an environment variable locally runs unchanged against Vault in CI. The seam between “which secret” and “where secrets come from” is exactly the seam that lets one test file move between environments without edits.

17.3 Secrets, RBAC, and the enterprise topology

In the enterprise topology, secret resolution happens entirely within the customer's network, because the backend runs there and Vault or the cloud secret manager is the customer's own. The author's RBAC claims, carried in the session token described in the security section, govern which secret paths the runner may resolve: a developer without permission to read a production signing key cannot resolve a reference to it, and the failure is reported at resolution time as an environment error with a clear authorisation message, not as a mysterious downstream fault. This makes the secret model a participant in the platform's overall least-privilege posture rather than a bypass around it.

Why this is non-negotiable for v1

Of the operational concerns in this part of the document, secrets management is the one that cannot be deferred, because its absence is not a missing convenience but an active hazard: without a sanctioned mechanism, every team invents an unsafe one. The ${secret:…} syntax with environment-variable and Vault sources is therefore part of the first release, not a later enhancement.

18. Principal Engineering Risks and Mitigations

A blueprint that lists only its strengths is not trustworthy. The risks below are the ones most likely to threaten delivery or operation; each is paired with the mitigation already embedded in the architecture and with the residual exposure that remains.

Risk	Mitigation in the architecture	Residual exposure
Dynamic-assembly memory growth in long suites.	Compile-once plus collectible AssemblyLoadContext with explicit unload; memory is asserted before and after each scenario.	A stray static reference can still pin a context; needs a leak-detection test in CI.
Untrusted customer DLLs run in the test process.	Bounded load context, least-privilege process, minimal global context.	Full sandboxing design deferred to a dedicated security review.
Ryuk disabled in locked-down enterprise clusters.	Fallback to .NET process-exit hooks for cleanup.	A hard SIGKILL can still orphan resources; needs a periodic sweep job.
Network partition produces false-positive failures.	Halt-and-reconnect on the client; TTL grace period on Ryuk; “inconclusive” verdict.	A long outage still ends the run; tunnel stability must be monitored.
LAB consumption mis-metered, eroding margin.	Per-session consumption telemetry treated as a functional requirement.	Metering accuracy needs independent reconciliation against cloud-provider billing.
Topological parity silently breaks as features are added.	Single typed contract between layers; same code path for local and remote endpoints.	Requires a parity test matrix that runs every suite on all three topologies.
.NET 8 LTS / Aspire / MAF are recent and still evolving.	Architecture isolates each behind a narrow internal interface.	Upstream breaking changes may force refactors; track preview releases closely.
Community provider quality varies, eroding user trust in the platform.	Two-tier model (Core, Community) with the maintainer-awarded Vouched badge and clear labelling; Vouched status requires review and an integration test in the official matrix.	A poorly behaved Community provider can still damage the platform's reputation by association; the registry surfaces Vouched status in error messages and documentation.
A provider that handles credentials leaks them into logs or stack traces.	Engine-level log redaction; mandatory security checklist for the Vouched badge covering credential lifetime, redaction, and TLS defaults.	An unbadged credential-handling Community provider is not gated on review; the trust model is documented and the install path makes the status visible.
Provider sprawl produces a catalogue users cannot navigate.	Curated index of Community providers; canonical dotted naming; the editor surfaces only registered providers in autocomplete.	Without active curation the index will accumulate stale or abandoned entries; an annual review is needed.
The structured event stream evolves and breaks downstream renderers or external consumers.	Schema-versioned events; renderers tolerate unknown fields; a deprecation window applies before any event-type removal; the schema is published alongside the engine and an integration test asserts compatibility with prior versions.	A determined external consumer that depends on undocumented fields can still break; the published schema is the only contract supported.
Report rendering quality varies across providers because diff rendering is provider-supplied.	An IStepDiffRenderer default exists in the engine that produces a generic structured diff; provider-supplied renderers are reviewed at Vouched badge award against a corpus of representative failures.	An unbadged Community provider with a poor renderer produces less useful failure reports; this is documented in the tier guarantees.
Teams seed test data ad hoc, producing brittle, mutually incompatible suites.	A first-class declarative seed block as the default strategy, applied inside the health-gated lifecycle; programmatic factories and snapshots offered as explicit alternatives.	The default cannot cover every case; teams with exotic data needs still write custom seeding, and the snapshot pipeline is post-MVP.
At scale the runner becomes a bottleneck: no selection, no safe parallelism, flaky shared state.	A composable selection language over existing metadata; scenario-level parallelism bounded by a concurrency budget; an explicit isolation contract defaulting to per-scenario reset and private topologies for parallel runs.	Aggressive parallelism still pressures developer hardware; the concurrency budget must be tuned per environment.
Secrets are hard-coded into YAML and leak into source control or reports.	A ${secret:…} reference syntax resolved at runtime from pluggable sources; report-layer redaction of any secret-derived value; RBAC-governed resolution in the enterprise topology.	A determined author can still place a literal secret in a non-secret field; a lint rule to detect secret-shaped literals is a fast follow.

19. Appendix: Technology Reference

This appendix consolidates the concrete technologies named in the blueprint, so an engineer can see at a glance which library or component owns each responsibility.

Component	Role in the platform
.NET 8 LTS	The runtime baseline. The engine targets .NET 8 LTS so adopter organisations on the current LTS can use the platform without changing runtime.
.NET Aspire	Orchestrates the application topology; provides service discovery, health-gated startup, and connection-string resolution.
DistributedApplication.CreateBuilder()	The programmatic Aspire builder API the engine uses to construct an AppHost from the YAML environment section. The headless runner passes DistributedApplicationOptions { DisableDashboard = true } because the dashboard requires environment variables that only the 'aspire run' tooling injects. AddContainer adds an image; AddProject(name, csprojPath) adds a project; AddPostgres / AddKafka / AddMongoDB / AddRedis / AddRabbitMQ / AddElasticsearch add typed managed resources. Each Add* call returns an IResourceBuilder that the runner retains so endpoints can be resolved through it (builder.GetEndpoint("http").Url) after StartAsync returns.
Testcontainers	Manages the lifecycle of dependency containers; supplies the k6 load-injection module and the Ryuk reaper.
Roslyn Scripting API	Compiles generated CSX into a reusable in-memory delegate (compile-once pattern).
Collectible AssemblyLoadContext	Isolates dynamically generated assemblies so they can be unloaded and reclaimed.
Respawn	Resets relational databases to a pristine state between tests, honouring foreign-key order.
System.CommandLine	Argument parsing and command dispatch for the headless vouchfx CLI runner (vouchfx run and the tag, owner, path, change-set, and prior-verdict selection flags). Pinned to 2.0 (GA).
Polly	Provides the bounded exponential-backoff resilience policies behind VERIFYMODE: RETRY.
System.Threading.Channels	Bounded producer/consumer queues giving backpressure in the Performance LAB.
Confluent Kafka client	Producer and consumer delegates with Avro and schema-registry support.
Confluent.SchemaRegistry	Subject registration and schema governance for Avro-published messages; version-locked with Confluent.Kafka (2.14.2).
Confluent.SchemaRegistry.Serdes.Avro	Serialiser/deserialiser adapters integrating Avro publishing/consuming with the schema registry; version-matched with Confluent.Kafka (2.14.2).
Apache.Avro	The Apache Avro codec; pinned to the exact version transitively required by Confluent.SchemaRegistry.Serdes.Avro (1.12.1) to prevent version downgrades under strict compilation flags.
Microsoft Agent Framework	Hosts the Planner, Generator, and Healer agents and their graph-based workflows.
grafana/k6	Containerised load generator orchestrated by the Performance LAB.
SSH tunnelling / reverse port forwarding	Carries traffic between the local runner and the cloud fabric, including inbound webhooks.
SAML 2.0 / OpenID Connect	Enterprise identity integration feeding RBAC into the CLI and IDE.
StepKindRegistry	Source-level registry of step providers, frozen at engine startup and indexed by family-and-provider name; powers the resolve stage of compilation.
[StepProvider] attribute / IProviderModule	Markers the registry uses to discover provider types at startup via reflection over referenced assemblies. There is no runtime plugin loading.
Schema fragments (per provider)	Each provider contributes a JSON Schema fragment for its step kind; the engine merges them into the unified schema consumed by the editor and the documentation generator.
Structured event stream	JSON Lines, schema-versioned event stream emitted by the engine during a run; the single substrate every renderer and the Healer agent consume.
Verdict taxonomy	The four-state classification of every step and scenario outcome — pass, fail, environment error, inconclusive — surfaced separately by the renderers.
Polling timeline	The per-attempt rendering of a RETRY step's history, derived directly from step-attempt events in the structured stream.
Reproducibility envelope	The full context bundled into every report — engine version, schema version, provider versions, image digests, content hashes, seeded variables — that lets a reader reproduce the exact run elsewhere.
IStepDiffRenderer	Optional provider interface (bool CanRender(JsonElement observation) and string? RenderDiff(JsonElement observation)) that supplies technology-specific rendering for the expected-versus-observed diff in the report. Renders LF-separated lines ready for terminal output; the reporting layer stays decoupled from provider-specific observation types.
seed block	Environment-level declaration of reference SQL files, document fixtures, and broker warm-up messages applied after the topology is healthy and before the first step runs.
Respawn (seeding context)	Already listed for reset; in the data lifecycle it provides the between-scenario clean baseline that the isolation contract depends on.
Test runner selection language	Composable selection over tag, ownership, path, change-set, and prior verdict, evaluated against metadata the author already writes.
maxParallel / concurrency budget	Operator-set bound on how many scenarios run concurrently, each against its own topology; parallelism is across scenarios, never within one.
${secret:source/path} reference	Runtime-resolved secret reference; the literal value never enters source control, logs, or reports, and the source (env, file, vault, cloud) is chosen by configuration.
Test doubles (WireMock / Mountebank)	Provisioned as ordinary Testcontainers when a real dependency cannot be used; the platform ships no built-in mocking, keeping doubles a visible, opt-in exception.

— End of the Technical Architecture & Engineering Blueprint. The companion document is the YAML DSL Specification & VSCode Extension Design.