YAML DSL Specification & VSCode Extension Design

The declarative testing language and its intelligent authoring environment

Language & Tooling Specification

Document	DSL Specification & VSCode Extension Design
Platform	E2E Microservices Integration Testing SaaS
Version	1.0 (Draft for Review)
Status	For engineering review
Audience	Language designers, extension developers, test authors
Companion	Technical Architecture & Engineering Blueprint
Date	May 2026

---

[Table of contents auto-generated by viewer; see section list at top of repo.]

1. Overview and Objectives

This document specifies two tightly related artefacts: the YAML Domain-Specific Language in which integration tests are written, and the Visual Studio Code extension that makes authoring that language productive. The two are treated together because a declarative DSL is only as good as the tooling around it. A language that is concise to read but offers no autocomplete, no validation, and no way to discover its own features will frustrate the developers it was meant to help. The specification therefore moves deliberately from the shape of the language to the experience of writing it.

The platform's central architectural decision — compiling declarative YAML into executable C# through Roslyn — is described in the companion Technical Architecture & Engineering Blueprint. This document assumes that mechanism and concentrates on the surface the test author actually touches: the grammar of the YAML, the meaning of each construct, the JSON Schema that validates it, and the editor features that turn a plain text file into a guided authoring experience.

1.1 Why a declarative DSL

Modern testing must bridge two audiences who rarely share a toolchain: backend engineers who think in code, and quality-assurance engineers and product analysts who think in scenarios. Declarative languages built on YAML and JSON have become the standard vehicle for that bridge because they let a complex, multi-stage scenario be expressed with very little boilerplate. The platform follows that consensus, but adds a deliberate escape hatch: when declarative syntax becomes awkward, the author can drop into an embedded C# script block and regain the full power of a programming language. The DSL's job is to make the common case effortless without making the hard case impossible.

1.2 Design objectives for the language

Five objectives govern every decision in the grammar. They are listed in priority order so that, where two objectives pull against each other, the reader knows which one wins.

1. Readability before brevity. A test should be understandable by someone who did not write it. Where a slightly longer keyword reads better than a terse symbol, the longer keyword is chosen.
2. Validity caught early. Structural mistakes must surface in the editor, before any container starts, through a strict JSON Schema.
3. A graceful escape hatch. Embedded C# must feel like a natural extension of the YAML, not a foreign body bolted on.
4. Determinism by default. Constructs that depend on timing — waiting for an event, polling a row — must be first-class, so authors never hand-roll sleeps.
5. Discoverability. Every capability of the language must be reachable through editor autocomplete; nothing important should live only in documentation.

2. DSL Design Principles

Before the grammar itself, it helps to fix a few principles that recur throughout the specification. They explain the shape of the language and prevent the grammar from reading as an arbitrary collection of keywords.

2.1 A test is a sequence of steps over shared state

Every test file describes an ordered list of steps. Steps execute top to bottom, and they communicate through a single shared object: a mutable dictionary of variables. A step may write into that dictionary — for example, by capturing an identifier from an HTTP response — and a later step may read from it. This is the same ScriptGlobalVariables context that the architecture document describes; from the language's point of view it is simply “the variables”. Modelling a test as steps over shared state is what allows a scenario to thread a generated identifier through a REST call, a Kafka event, and a database assertion without the author ever managing wiring by hand.

2.2 Declarative first, imperative on request

The grammar offers a declarative form for every common operation: sending a request, publishing a message, asserting a row. Each declarative step compiles to optimised C#. When the author needs logic the declarative form cannot express — a loop, a conditional, a non-trivial transformation — they use a script step containing C#. The two forms are interleaved freely, and they share the same variable context, so moving a value from a declarative step into a script step requires no ceremony.

2.3 The engine owns timing, not the author

Distributed systems are asynchronous, so a test frequently must wait for something to become true. The language never asks the author to express that wait as a sleep. Instead it provides a verification mode — written as verifyMode: RETRY — that instructs the engine to poll, with bounded exponential backoff, until the assertion holds or a timeout expires. The author declares the intent (“this should eventually be true”); the engine owns the mechanics.

2.4 One schema, many uses

A single JSON Schema defines what a valid test file may contain. That schema is the contract used by the compiler to reject malformed input, by the VSCode extension to provide autocomplete and inline validation, and by the documentation generator. Because all three derive from one source, the editor can never advertise a construct the compiler does not understand. Section 8 specifies the schema in detail.

3. YAML Document Structure

This section defines the top-level shape of a test file. A file carries the conventional extension .e2e.yaml, which is also the pattern the VSCode extension uses to recognise it. The file has four top-level sections, and only the steps section is mandatory.

Top-level key	Required	Purpose
metadata	No	Human-facing description of the test — name, owner, tags — used in reporting and for filtering which suites a CI run executes.
environment	No	Declares the dependencies the test needs (databases, brokers, services). If omitted, the test runs against the ambient Aspire topology.
variables	No	Seeds the shared variable context with constant values before the first step runs.
steps	Yes	The ordered list of actions and assertions that constitute the test.

Table 3.1 — The four top-level sections of an .e2e.yaml file.

3.1 The metadata section

The metadata section carries information about the test rather than instructions to the engine. Its fields feed reporting dashboards and let the runner select a subset of tests — for example, only those tagged as a smoke test, or only those owned by a particular team. None of it affects execution.

metadata:
  name: "User registration propagates to billing"
  owner: "payments-team"
  tags: [smoke, billing, kafka]
  description: >
    Verifies that creating a user via the REST API results in a
    BillingAccountCreated event and a materialised MongoDB document.

The owner and tags fields are not decoration: the test runner's selection language filters on them directly, so a CI job can request “tagged regression, owned by payments” in a single invocation. Tagging discipline at authoring time is therefore what makes large suites navigable later. The runner's full selection model — by tag, ownership, path, change-set, and prior verdict — is described in Section 16 of the companion Technical Architecture & Engineering Blueprint.

3.2 The environment section

The environment section names the infrastructure the test depends on. Each entry maps a logical name — the name the steps will use — to a resource specification. The orchestration layer reads this section to decide which containers to provision and in what order, and it is also the information the VSCode extension cross-references to know which logical names are valid elsewhere in the file. Declaring a dependency here is what makes orders-db a resolvable target in a later database assertion.

Both services and dependencies must have a mapping (YAML { … }) value; a bare scalar (e.g. an unquoted service name or type) is malformed and rejected by the parser at load time with a line/column error.

Services are the system under test: the customer's own code that the suite exercises. Each service is brought to the topology in one of two forms. The image form references a container image — the customer's CI has already built and pushed it — and is the recommended default for speed and isolation. The project form references a csproj path — the engine builds and runs the project as part of suite startup — and is the convenience for teams iterating on a service locally. Use exactly one of the two fields per service. Dependencies, by contrast, are managed resources Aspire knows how to provision (databases, brokers, caches): they declare a type and the engine selects the appropriate image.

environment:
  services:
    orders-api:
      # Recommended: image form. Pulled from any OCI registry.
      image: myorg/orders-api:1.2.3
      # Or: project form. The engine builds the csproj at suite start.
      # project: ./src/Orders.Api/Orders.Api.csproj
  dependencies:
    orders-db:    { type: postgres, version: "16" }
    events:       { type: kafka, schemaRegistry: true }
    search:       { type: elasticsearch }

3.2.1 Container images and registries

Image references in the environment section follow the standard OCI / Docker image format: [registry-host[:port]/]namespace/repository[:tag|@digest]. Un-prefixed names resolve through Docker Hub (so postgres:16 is the official Postgres image); fully-qualified names target any OCI-compliant registry without further configuration (GitHub Container Registry, Azure Container Registry, ECR, GCR, Harbor, Artifactory). Authentication to private registries is handled by the underlying container runtime: a docker login beforehand or a CI step that populates the credential store is what enables the pull, and the test platform inherits those credentials transparently.

Three controls on top of the image reference give authors finer-grained control. An imageRegistry override at the environment level prefixes every un-prefixed image with the given registry hostname, which is the form teams in regulated environments use to redirect public images to an internal mirror. An imagePullPolicy field accepts the standard Docker values — Always, Missing, Never — with the engine choosing a sensible default per image (Always for moving tags such as :latest, Missing for semver-shaped tags). Explicit digest pinning with @sha256:… is supported and is the recommended form for production-grade reproducibility, because tags can move under the team's feet whereas digests are byte-stable.

environment:
  imageRegistry: artifactory.mycompany.com/docker-mirror   # optional default
  services:
    orders-api:
      image: myorg/orders-api@sha256:9f4c…         # digest-pinned
      imagePullPolicy: Missing
  dependencies:
    orders-db: { type: postgres, version: "16" }
    events:    { type: kafka, schemaRegistry: true }

An image-pull failure — registry unreachable, authentication failed, image not found — is classified as an environment error per the verdict taxonomy, not a test failure. The report names the registry hostname and the authentication status alongside the runtime's underlying error, so an engineer can act on it without spelunking. This is the most common cause of early-pilot frustration in Docker-based testing tools, and the architecture handles it explicitly rather than letting it manifest as a generic timeout.

Note on imagePullPolicy enforcement: The imagePullPolicy field is currently accepted by the validator but not enforced by the runtime; the underlying Docker/container default applies (pull if image is missing locally). This field is reserved for future enforcement. Authors should not rely on it to prevent unexpected image pulls; use explicit @sha256:… digest pinning if reproducibility is critical.

3.2.2 Schema registries for message brokers

A Kafka broker can optionally be paired with a Confluent Schema Registry to provide schema governance and Avro serialisation. When Avro steps in the test need to publish or consume from a schema registry, declare the flag schemaRegistry: true on the Kafka dependency. The engine provisions the registry as a sidecar container and wires it to the broker; steps then reference the Kafka dependency by name to access the registry.

environment:
  dependencies:
    events:
      type: kafka
      schemaRegistry: true    # additionally provision a Confluent Schema Registry

Once declared with schemaRegistry: true, a mq-publish.kafka step can include an avro block to publish Avro-serialised messages, and an mq-expect.kafka step can include an avro block to consume and decode them (see §5.2 and §5.3). The registry URL is automatically staged and made available to these steps. No manual configuration of registry endpoints is required.

3.2.3 Seeding initial state

Resetting a database between tests is only half of the data lifecycle; the other half is establishing the state a test needs before it runs. An optional seed block inside the environment section declares that work declaratively. It can apply reference SQL files to a relational dependency, load document fixtures into a document store, and publish warm-up messages to a broker. The engine applies the seed after the topology is healthy and before the first step executes, so a step never races the data it depends on.

environment:
  dependencies:
    orders-db: { type: postgres }
    events:    { type: kafka }
  seed:
    orders-db:
      sql: [ "fixtures/reference-currencies.sql",
             "fixtures/reference-products.sql" ]
    events:
      publish:
        - topic: catalog.snapshot
          payload: { from: "fixtures/catalog.json" }

Declarative fixtures are the recommended default because they live in source control beside the test, are visible in review, and produce the same baseline on every machine. When data must be created through the system's own APIs rather than injected directly — because the creation path is itself part of what is under test — a script step early in the scenario is the right tool instead, constructing entities and capturing their identifiers for later steps. A seed that fails to apply produces an environment error, not an assertion failure, so a broken fixture is never mistaken for a broken system. The reproducibility envelope records the content hash of every applied fixture.

State reset between sequential scenarios: Between sequential scenarios that share a single topology, PostgreSQL, SQL Server, MySQL, MongoDB, Redis and Elasticsearch dependencies are automatically reset after each scenario completes. Each store type uses an appropriate data-clearing mechanism — relational tables are cleared via delete order (preserving structure), MongoDB collections are cleared per-collection (preserving indexes), Redis executes FLUSHDB against the designated database, and Elasticsearch executes a delete-by-query across open indices (preserving mappings and settings). Brokers (Kafka, RabbitMQ, NATS, Azure Service Bus) are not applicable — messages are consumed per step; DynamoDB and MinIO are not reset. A failed reset surfaces as an environment error naming the dependency — never as a test failure. Seed applies to the first scenario only; subsequent scenarios receive cleared data. Parallel scenarios (via --parallel) are unaffected — each receives its own topology and fresh containers, isolation by construction. The common patterns guide explains per-store reset mechanics.

3.2.4 Configuring the system under test

Services declared under environment.services may carry an optional env map to configure the service container with environment variables. Each variable name maps to a value that may be a literal string, zero or more ${conn:<dependency>} references to a dependency declared under environment.dependencies, or both interleaved. The engine resolves connection references at topology-build time in the consumer service's network context — that is, containerised services receive the container-internal hostnames and ports, not the host-published ones.

environment:
  services:
    orders-api:
      image: myorg/orders-api:latest
      env:
        DATABASE_URL: "postgresql://orders-db:5432/orders"
        MESSAGE_BROKER: "${conn:events}"
        CACHE_ENDPOINT: "${conn:cache.host}:${conn:cache.port}"
        DB_USERNAME: "${conn:orders-db.username}"
        WEBHOOK_CALLBACK: "https://webhook-service/callbacks"
  dependencies:
    orders-db:  { type: postgres, version: "16" }
    events:     { type: kafka }
    cache:      { type: redis }

Supported dependency kinds and their available parts are:

Kind	Supported parts
postgres, mysql, sqlserver, mongodb	host, port, username, password, database
kafka	host, port (internal bootstrap; scheme not included)
rabbitmq	host, port, username, password
redis	host, port, password
nats	host, port, username, password
elasticsearch	host, port
mailpit	host, port (SMTP endpoint)
dynamodb	host, port
minio	host, port

Unknown dependency names or parts result in a validation error before the topology is started, reported as Inconclusive — the test never ran, consistent with the §12.1 treatment of schema and secret-reference authoring failures. Secrets (${secret:…}) are not supported in environment values — secrets resolve at step-execution time, whereas container environment variables are baked in at startup, and exposing secret values via docker inspect would defeat the reproducibility envelope. A system under test that needs real credentials at runtime should obtain them through its own configuration mechanism; the managed-dependency credentials reachable through ${conn:…} are ephemeral, Aspire-generated test values, not §17 secrets. Literal braces in env values — JSON fragments, ${OTHER_VAR}-style self-expansion placeholders that are not ${conn:…} or ${secret:…} references — are passed through to the container verbatim.

Additionally, image-form services automatically receive the Docker host gateway alias (--add-host=host.docker.internal:host-gateway), allowing containerised services to reach listeners on the host. When a webhook listener variable is staged (see §5.5), it is made available to the consumer SUT both as {<listener>} (host loopback address, used by host-local steps) and as {<listener>_container} (the host-gateway form, suitable for passing to containerised services as a callback URL). Declaring two webhook listeners whose names collide through this aliasing (a listener named x alongside one named x_container) is a validation error; author-declared variables: follow the usual forward-only assignment rules and are not checked against the synthesised alias.

3.2.5 Test doubles

When a dependency cannot be exercised for real — a payment gateway that costs money, a third-party API with no sandbox, a service that does not yet exist — the platform's position is that a test double is provisioned like any other dependency: as a container running a stubbing tool such as WireMock or Mountebank, declared in this same section and addressed through the same logical name. The platform ships no built-in mocking; a double is a deliberate, visible entry in the environment rather than a hidden behaviour. Because the double speaks the real protocol, the steps that call it are written identically to steps that call the real service, and swapping one for the other later requires no change to the test logic — only to this declaration.

3.3 The variables section

The variables section pre-loads the shared context with constants before any step executes. It is the right place for fixed test data — a known tenant identifier, a base path, an expected currency code. Values defined here are visible to every step and can be referenced with the placeholder syntax defined in section 6.

variables:
  tenantId:  "acme-corp"
  basePath:  "/api/v1"
  currency:  "GBP"

3.4 The steps section

The steps section is the heart of the file: an ordered list, each entry of which is a single typed step. The remaining sections of this specification are largely an enumeration of the step types and what each one means. Every step shares a small set of common fields, defined next, and then adds fields specific to its type.

4. Common Step Fields

Every step, regardless of type, may carry the fields below. Defining them once here keeps the per-type sections focused on what is genuinely distinctive about each step.

Field	Required	Meaning
id	Yes	A unique identifier for the step within the file. Used in reporting and as the anchor for any failure message.
type	Yes	The kind of step, in dotted family.provider form, e.g. http.rest or db-assert.postgres. Determines which additional fields are valid. A bare family name (with no provider) is not accepted.
description	No	A short human-readable explanation shown in test output.
capture	No	A map of variable names to extractor expressions; writes values from this step's result into the shared context (see section 6).
verifyMode	No	Either IMMEDIATE (the default) or RETRY. RETRY instructs the engine to poll until the step's assertions hold or a timeout expires (see section 7).
timeout	No	An upper bound on how long the step may take, expressed as a duration such as 30s — enforced for every verify mode. An IMMEDIATE step that exceeds it resolves as Inconclusive (step-timeout, never Fail — §12.1): providers observe the step's cancellation cooperatively, and a body that completes past the bound has its outcome superseded. Where the provider's emitted body sets a built-in transport timeout (the HTTP, AWS and SQL command-timeout conventions), a declared value replaces it as the governing bound, so a budget longer than that convention is honoured; when omitted, those conventions (typically 30 seconds) remain the de facto bound. For a RETRY step it bounds the polling window.
continueOnFailure	No	When true, a failed assertion in this step is recorded but does not abort the remaining steps. Defaults to false.

Table 4.1 — Fields available on every step type.

5. Step Types and the Provider Model

This section defines the language's step types. The subsections that follow describe the shipped families with worked YAML examples. One further family — rpc — is recognised by the language and reserved in the schema but has no Core provider; it is covered alongside the others in subsection 5.7, where the provider model is introduced and the full community catalogue is tabulated. The dotted naming convention used throughout this section is best understood once that model is on the table.

The vocabulary used here is precise. A family names what is being done at the level of intent: http issues a request and asserts on a response; db-assert queries a store and asserts on the result. A provider names the concrete technology that implements the family: http.rest is a REST-over-HTTP provider; db-assert.postgres is a PostgreSQL provider. An author always writes the dotted form — type: db-assert.postgres — there is no bare-family shorthand, even for a family with only one registered provider today: a family gaining a second provider later must never change the meaning of a step type an existing file already uses. The editor and error messages likewise only ever render the dotted name. The full provider model, including how new providers are added by community contribution at the source level, is described in section 5.7 below and at architectural depth in section 13 of the companion Technical Architecture & Engineering Blueprint.

Together the eleven shipped families are sufficient to express the platform's reference scenario — a transaction crossing REST, Kafka, a database, and a webhook — end to end, and the provider model below makes each family extensible to whichever concrete technology a given microservices estate actually uses. The rpc family is reserved for the gRPC and similar typed-RPC scenarios that the community catalogue will fill in post-MVP.

5.1 The http family

An http step issues an HTTP request to one of the services declared in the environment section and asserts properties of the response. The target field names a logical service rather than a URL, so the engine can resolve the real address through Aspire service discovery — the same test therefore works whether the service runs locally or in the cloud. The family now has two Core providers: http.rest (the example below) and http.soap (§5.1's second subsection, below). http.graphql remains a planned future addition (section 5.7). Like every family, the dotted form is always required — this was already true before http.soap existed and remains unchanged now that it does.

Field	Meaning
target	Logical name of the service to call, as declared under environment.services.
method	The HTTP verb: GET, POST, PUT, PATCH, or DELETE.
path	The request path, which may contain {placeholder} and ${secret:…} tokens resolved at execution time.
headers	An optional map of request headers. Each header value may contain {placeholder} and ${secret:…} tokens.
body	An optional request body. A YAML scalar is treated as a template string; a YAML mapping or sequence is serialised to JSON and then used as a template. All {placeholder} and ${secret:…} tokens are resolved at step-execution time.
expect	An assertion block: expected status code, and optional assertions on response headers and body fields.

- id: create-user
  type: http.rest
  description: "Register a new user"
  target: orders-api
  method: POST
  path: "{basePath}/users"
  body:
    email: "jane@{tenantId}.example"
    plan:  "standard"
  expect:
    status: 201
    body:
      id: { exists: true }
  capture:
    newUserId: "$.id"        # JSONPath into the response body

http.soap — step fields

http.soap calls a SOAP 1.1 service. It is the RAW-ENVELOPE provider (a deliberate v1 design choice): the author supplies the complete SOAP envelope XML themselves — there is no auto-wrapping of a bare payload into a <soap:Envelope> skeleton. This keeps the provider's contract small and unambiguous (no guessing at namespace prefixes, SOAP version, or header placement) at the cost of a little authoring verbosity.

Field	Required	Meaning
target	Yes	Logical name of the service to call, as declared under environment.services. Identical addressing to http.rest.
path	Yes	The request path; may contain {placeholder} and ${secret:...} tokens.
action	No	The SOAPAction header value (the SOAP 1.1 convention). Sent quoted per the SOAP 1.1 specification (SOAPAction: "value"). May contain {placeholder} and ${secret:source/path} tokens. Omitted entirely when not declared.
envelope	Yes	The full SOAP request envelope XML, as a raw template string. Sent as Content-Type: text/xml; charset=utf-8 (SOAP 1.1). SOAP 1.2's application/soap+xml (with the action folded into the Content-Type rather than a separate header) is a documented v1 limitation, not supported. May contain {placeholder} and ${secret:...} tokens, resolved at step-execution time exactly like http.rest's body.
expect.status	No	Expected HTTP status code. Defaults to 200 when omitted.
expect.fault	No	Whether the response is expected to contain a SOAP Fault element (matched by local-name()='Fault', recognising both the SOAP 1.1 and SOAP 1.2 envelope namespaces). Defaults to false.
expect.xpath	No	A list of {path, value} assertions evaluated against the response envelope, after the status and fault-expectation checks both pass.

Fault/status precedence. A SOAP Fault is a first-class, expected outcome, not merely a transport failure: fault-expectation is checked first (a mismatch — either an unexpected Fault present, or an expected Fault absent — is an immediate Fail); only once fault-expectation is satisfied does the step check the HTTP status; only once both pass are any declared expect.xpath assertions evaluated. On an unexpected fault the observation carries only the faultcode (a fixed QName-shaped token, e.g. soap:Server — safe to report verbatim) and never the faultstring, which may embed SUT-originated data.

XPath namespace handling. No namespace manager is registered — exactly like http.rest's own XPath capture, for consistency. Write namespace-agnostic expressions using local-name() (e.g. //*[local-name()='Body']/*[local-name()='GetUserResponse']); a prefixed expression (//soap:Body) is not guaranteed to resolve.

Capture. capture: supports XPath expressions only (there is no JSON body to JSONPath over) — declare the explicit { xpath: … } form; a bare (JSONPath-default) capture entry against a SOAP response is always unmet.

Example (a successful call with an XPath assertion and capture, and a SOAP Fault treated as an expected outcome):

- id: get-user
  type: http.soap
  target: app
  path: /soap/users
  action: "http://example.com/GetUser"
  envelope: |
    <?xml version="1.0" encoding="utf-8"?>
    <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
      <soap:Body>
        <GetUser xmlns="http://example.com/users">
          <UserId>{userId}</UserId>
        </GetUser>
      </soap:Body>
    </soap:Envelope>
  expect:
    status: 200
    xpath:
      - path: //*[local-name()='Body']/*[local-name()='GetUserResponse']/*[local-name()='Status']
        value: "OK"
  capture:
    userName:
      xpath: //*[local-name()='Body']/*[local-name()='GetUserResponse']/*[local-name()='Name']

- id: get-missing-user
  type: http.soap
  target: app
  path: /soap/users
  action: "http://example.com/GetUser"
  envelope: |
    <?xml version="1.0" encoding="utf-8"?>
    <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
      <soap:Body>
        <GetUser xmlns="http://example.com/users">
          <UserId>does-not-exist</UserId>
        </GetUser>
      </soap:Body>
    </soap:Envelope>
  expect:
    status: 500
    fault: true

5.2 The mq-publish family

A mq-publish step produces a message onto a broker. It is most often used to drive a test by injecting an event that the system under test is expected to react to. The mq-publish family has five Core providers: mq-publish.kafka (with full schema-registry and Avro support), mq-publish.rabbitmq, mq-publish.nats, mq-publish.azureservicebus, and mq-publish.redis (Redis Streams). The dotted form is always required; a bare type: mq-publish is not a valid step type.

5.2.1 mq-publish.kafka: plain payload

The Kafka provider ships with two distinct paths. The plain-payload path publishes a UTF-8 string (a literal value or inline JSON) as the message value. The field set is as follows:

Field	Required	Meaning
target	Yes	Logical name of the kafka dependency declared under environment.dependencies.
topic	Yes	The Kafka topic to publish to. May contain {placeholder} and ${secret:...} tokens.
key	No	Optional message key. May contain {placeholder} and ${secret:...} tokens.
payload	Yes	The message body as a UTF-8 string (literal or inline JSON). May contain {placeholder} and ${secret:...} tokens.
headers	No	Optional map of message-header names to their string values. Each value may contain {placeholder} and ${secret:...} tokens.

Example (plain publish):

- id: publish-order
  type: mq-publish.kafka
  target: events
  topic: orders.created
  key: "{orderId}"
  payload: |
    {
      "orderId": "{orderId}",
      "customerId": "{customerId}",
      "amount": 99.99
    }
  headers:
    x-trace-id: "{traceId}"

5.2.2 mq-publish.kafka: Avro with schema registry

The Avro path publishes a message whose value is built as an Avro GenericRecord from an inline Avro schema and a field map, and serialises it through a provisioned Confluent Schema Registry. The schema is auto-registered under the declared subject. When the avro block is present, the plain payload field is ignored.

Field	Location	Required	Meaning
avro	root	No	When present, switches to the Avro path. Absent → plain path.
schemaRegistry	avro	Yes	Logical name of the kafka dependency whose schema registry to publish through (its schemaRegistry-enabled registry URL is provisioned under environment.dependencies).
subject	avro	Yes	The schema-registry subject under which the inline schema is registered.
schema	avro	Yes	The inline Avro schema as avsc JSON. Must be a record schema.
record	avro	Yes	Map of Avro field names to their values. Each value may contain {placeholder} and ${secret:...} tokens; field names are used verbatim. Values are coerced to the schema's primitive field types at execution time.

Example (Avro publish):

- id: publish-billing-event
  type: mq-publish.kafka
  target: events
  topic: billing.events
  avro:
    schemaRegistry: events
    subject: billing-value
    schema: |
      {
        "type": "record",
        "name": "BillingEvent",
        "namespace": "com.example",
        "fields": [
          { "name": "userId", "type": "string" },
          { "name": "amount", "type": "double" }
        ]
      }
    record:
      userId: "{newUserId}"
      amount: "99.99"

The mq-publish.kafka provider auto-registers the supplied inline schema under the declared subject via the Confluent AvroSerializer's default auto-registration at publish time. No subject-naming-strategy configuration, compatibility-level options, or ephemeral-vs-non-ephemeral distinction are currently implemented.

mq-publish.redis — step fields

Publishes one UTF-8 message to a Redis Stream via XADD, carried under the family's canonical payload stream field. Unlike mq-publish.nats (subject + optional derived JetStream stream name), a Redis Stream key IS the addressing unit — there is no separate "subject" concept and no stream-name derivation: the author names the stream directly, and XADD creates it automatically when it does not yet exist. A Pass verdict confirms the entry was appended to the stream; it does not confirm that a consumer has read it — verify with a following mq-expect.redis step.

The payload-field convention (fixed in v1, not configurable). The message is written as a single stream field literally named payload — this is the field name mq-expect.redis reads back. A field name mismatch is only possible if you hand-write a different producer against the same stream; a configurable field name is a possible additive v1.x extension.

Field	Required	Meaning
target	Yes	Logical name of the redis dependency to publish to, as declared under environment.dependencies.
stream	Yes	The Redis Stream key to XADD to. May contain {placeholder} and ${secret:...} tokens.
payload	Yes	The message body as a UTF-8 string (literal or inline JSON), written as the value of the canonical payload stream field. May contain {placeholder} and ${secret:...} tokens.

Example:

- id: publish-order-event
  type: mq-publish.redis
  target: cache
  stream: orders-created
  payload: '{"event":"order-created","orderId":"ord-001"}'

5.3 The mq-expect family

An mq-expect step consumes from a broker and asserts that a matching message arrives. Because the message may not be present the instant the step runs, this step is almost always paired with verifyMode: RETRY: the engine polls the source, with backoff, until a message satisfying the match block appears or the timeout expires. The mq-expect family has five Core providers: mq-expect.kafka, mq-expect.rabbitmq, mq-expect.nats, mq-expect.azureservicebus, and mq-expect.redis (Redis Streams). The dotted form is always required; a bare type: mq-expect is not a valid step type.

5.3.1 mq-expect.kafka: plain payload

The Kafka provider ships with two distinct paths. The plain-payload path evaluates assertion criteria against the UTF-8 message value (a literal string or inline JSON). At least one criterion must be declared.

Field	Required	Meaning
target	Yes	Logical name of the kafka dependency to consume from, as declared under environment.dependencies.
topic	Yes	The Kafka topic to consume the message from.
match	Yes	An assertion-criteria block (see below).
match.key	No	Expected message key (ordinal string equality). May contain {placeholder} and ${secret:...} tokens.
match.headers	No	Optional map of expected header names to their expected string values. Each value may contain {placeholder} and ${secret:...} tokens.
match.payloadContains	No	Optional substring that the UTF-8 message value must contain (ordinal). May contain {placeholder} and ${secret:...} tokens.
match.json	No	Optional map of JSONPath expressions to their expected string values. The message value is parsed as JSON; each JSONPath must select a node whose stringified value equals the expected value. Each expected value may contain {placeholder} and ${secret:...} tokens.

Example (plain expect with RETRY):

- id: expect-billing-event
  type: mq-expect.kafka
  target: events
  topic: billing.account.created
  verifyMode: RETRY
  timeout: 30s
  match:
    key: "{newUserId}"
    json:
      $.userId: "{newUserId}"
      $.status: "PENDING"
  capture:
    billingAccountId: "$.accountId"

5.3.2 mq-expect.kafka: Avro with schema registry

The Avro path decodes the consumed message as an Avro GenericRecord (fetching the writer schema from the registry by the message's embedded schema id), converts it to a JSON string, and evaluates the same match criteria against that JSON. The decoded JSON is transparent to the author: they write the same match criteria as they would for a plain JSON payload.

Field	Location	Required	Meaning
avro	root	No	When present, switches to the Avro path. Absent → plain path.
schemaRegistry	avro	Yes	Logical name of the kafka dependency whose schema registry to decode through (its schemaRegistry-enabled registry URL is provisioned under environment.dependencies).
subject	avro	No	Optional schema-registry subject (informational; the writer schema is fetched by the message's embedded schema id).
schema	avro	No	Optional inline reader schema as avsc JSON (informational for the expect side).

Example (Avro expect with RETRY):

- id: expect-billing-avro-event
  type: mq-expect.kafka
  target: events
  topic: billing.events
  verifyMode: RETRY
  timeout: 30s
  avro:
    schemaRegistry: events
  match:
    json:
      $.userId: "{newUserId}"
      $.amount: "99.99"

mq-expect.redis — step fields

Asserts that a message matching the declared criteria is present on a Redis Stream. The step scans the stream via XRANGE <stream> - + COUNT 10000 on every attempt (mirroring mq-expect.nats's retained-log rationale and its MaxMsgs=10000 bound exactly), so the check is idempotent under verifyMode: RETRY. As with mq-expect.nats, do NOT share a single redis dependency across scenarios that assert on the same stream — entries from a prior run will produce a false Pass; use distinct stream keys per scenario.

Scan bound. If a stream retains more than 10 000 entries, only the first 10 000 (oldest first, from the start of the stream) are inspected on a given attempt — an entry beyond that offset is not found. MAXLEN-trim the stream, or keep the asserted entry within the first 10 000, if this matters for your scenario.

The payload-field convention (fixed in v1, not configurable). Only entries carrying a field literally named payload are matched — the exact convention mq-publish.redis writes. An entry produced under any other field name is silently excluded from matching; on a Fail, the observation reports scanned (total entries inspected) and lackingPayloadField (how many of those lacked the field), so a SUT that writes under a different field name has a concrete diagnostic rather than a bare matched:false.

Field	Required	Meaning
target	Yes	Logical name of the redis dependency to consume from, as declared under environment.dependencies.
stream	Yes	The Redis Stream key to scan via XRANGE.
match	Yes	An assertion-criteria block (see below). At least one criterion must be declared.
match.payloadContains	No	Optional substring that the UTF-8 message payload must contain (ordinal). Only evaluated against entries carrying the payload field. May contain {placeholder} and ${secret:...} tokens.
match.json	No	Optional map of JSONPath expressions to their expected string values. The message payload is parsed as JSON; each JSONPath must select a node whose stringified value equals the expected value. Each expected value may contain {placeholder} and ${secret:...} tokens.

Example (with RETRY):

- id: assert-order-event
  type: mq-expect.redis
  target: cache
  stream: orders-created
  verifyMode: RETRY
  timeout: 30s
  match:
    payloadContains: "order-created"
    json:
      $.orderId: ord-001

5.4 The db-assert family

A db-assert step queries a data store and asserts properties of the result. This is the family where the value of the provider model is most visible: PostgreSQL, SQL Server, MySQL, MongoDB, and DynamoDB all have genuinely different query languages and assertion semantics, and each is implemented by its own provider with its own schema fragment. Like every family, the bare type: db-assert form is not accepted; the author must always choose a provider — db-assert.postgres, db-assert.mongodb, and so on. Like mq-expect, db-assert steps are commonly run with verifyMode: RETRY because a materialised view may take a moment to update.

The example below uses db-assert.postgres. It takes a SQL string with a parameter dictionary; the equivalent db-assert.sqlserver and db-assert.mysql steps look almost identical, db-assert.mongodb supplies a collection name and a filter document instead, and db-assert.dynamodb a table name and key condition. (Elasticsearch and Redis assertions live in the separate cache-assert family — cache-assert.elasticsearch and cache-assert.redis.) Each provider's reference documentation describes its precise field set; the common fields below are shared by every provider in the family.

Field	Meaning
target	Logical name of the data store dependency.
query	The query to run, interpreted by the chosen provider — SQL for db-assert.postgres and other relational providers, a filter document for db-assert.mongodb, a query-DSL body for db-assert.elasticsearch, a key pattern for db-assert.redis.
expect	An assertion block: expected row or document count, and assertions on individual field values.

- id: assert-projection
  type: db-assert.postgres
  target: orders-db
  verifyMode: RETRY
  timeout: 20s
  query: |
    SELECT plan, billing_account_id
    FROM users
    WHERE id = @userId
  parameters:
    userId: "{newUserId}"
  expect:
    rowCount: 1
    row:
      plan: "standard"
      billing_account_id: "{billingAccountId}"

db-assert.dynamodb — step fields

Asserts against a DynamoDB item via a single GetItem call. The engine provisions a real dynamodb-local container for the declared dependency (type dynamodb); the connection is a plain container with no HTTP health path of its own — see §3.2's dynamodb row and the engine's own EnvironmentMapper documentation for the health-gating choice. key is a deliberately simplified flat JSON template, not the low-level DynamoDB wire form: each top-level value becomes an S/N/BOOL attribute at execution time (a JSON string, number, or boolean respectively — nested objects/arrays are rejected, since a primary key cannot contain them).

Field	Required	Description
target	Yes	The logical name of the dynamodb dependency declared in environment.dependencies.
table	Yes	The DynamoDB table name. Supports {placeholder} substitution.
key	Yes	A flat JSON object template naming the primary key (partition key, optionally plus a sort key), e.g. {"orderId":"{orderId}"}. May contain {placeholder} tokens inside JSON string values; a JSON-escape guard prevents a resolved value from breaking out of its enclosing string (the same hardening db-assert.mongodb's filter resolution applies).
expect.exists	No	Whether the item is expected to exist. Defaults to true when omitted.
expect.item	No	Map of flat (top-level) attribute name to expected string value, compared against the GetItem result (S as-is, N as the raw stored number string, BOOL as "true"/"false"). Nested attributes are not supported in v1. Mutually exclusive with expect.exists: false — an absent item has no fields to assert on.

Number canonicalisation caveat. DynamoDB canonicalises N values — it trims trailing zeros, so an item written as 1.0 is stored (and compared) as 1. Declare the canonical form in expect.item ("1", not "1.0"), or the comparison fails on a value that looks identical.

Substitution scope. db-assert.dynamodb fields resolve {placeholder} tokens only; ${secret:…} references are not resolved by this provider and pass through literally. This differs from storage-assert.s3, which resolves both.

Use verifyMode: RETRY when the item may lag the transaction that writes it (the same post-stimulus convergence gap db-assert.postgres/db-assert.mongodb already accommodate) — GetItem is read-only and idempotent, so every RETRY poll attempt is safe to repeat.

- id: assert-order-written
  type: db-assert.dynamodb
  target: orders-db
  verifyMode: RETRY
  timeout: 15s
  table: Orders
  key: '{"orderId":"{orderId}"}'
  expect:
    exists: true
    item:
      status: "SHIPPED"

5.5 The webhook-listen family

A webhook-listen step opens an ephemeral local HTTP listener and asserts that the system under test calls it. This is how the platform validates outbound, asynchronous notifications. In the cloud topology, the architecture's reverse SSH port forwarding ensures that a webhook fired by a cloud-hosted service still reaches this listener on the developer's machine. The step is inherently asynchronous and is therefore always combined with verifyMode: RETRY.

The Core provider is webhook-listen.http. The engine stands up an ephemeral host-owned HTTP listener for each declared listener name and stages its bound URL in two forms: {<listener>} (loopback, for host-local steps) and {<listener>_container} (host-gateway form, for containerised SUTs to use as a callback URL). Both are available in Vars so authors can interpolate them into callback registration calls in earlier steps. At execution time the step asserts that at least one captured inbound HTTP request satisfies all declared match criteria.

Field	Meaning
listener	The logical listener name whose URL is staged at svc::<listener> and Vars[<listener>] for earlier steps to reference. Must not be empty.
match	The criteria an inbound request must satisfy. At least one criterion must be declared.

The match block contains the following optional criteria (all conjunctive — every declared criterion must hold):

Criterion	Meaning
method	Expected HTTP method (e.g. "POST"), matched case-insensitively. May contain {placeholder} and ${secret:…} tokens.
path	Expected request path (token-stripped, SUT-facing). Matched by exact ordinal equality; includes any query string. May contain {placeholder} and ${secret:…} tokens.
headers	Optional map of expected header names to their string values. Names are case-insensitive; values are ordinal. Each value may contain {placeholder} and ${secret:…} tokens.
bodyContains	Optional substring the request body must contain (ordinal). May contain {placeholder} and ${secret:…} tokens.

Sharp edge: The captured path includes the query string, so a callback to /cb?x=1 is not matched by path: /cb — the query string must be included in the path criterion.

Example: A service publishes a webhook callback URL to a third-party system, which then fires the callback asynchronously. Earlier steps thread the callback URL via placeholders.

- id: register-webhook
  type: http.rest
  target: third-party-api
  method: POST
  path: "/register"
  body:
    callbackUrl: "{webhook-listener}"
  expect:
    status: 200

- id: wait-for-webhook
  type: webhook-listen.http
  verifyMode: RETRY
  timeout: 30s
  listener: webhook-listener
  match:
    method: POST
    path: "/callbacks/delivery"
    headers:
      x-signature: "expected-hmac"
    bodyContains: "order-confirmed"
  capture:
    deliveryId: "$.id"        # JSONPath into the captured request body

5.6 The script family

A script step is the escape hatch. It contains a block of code in a chosen language — the Core provider is script.csharp, the embedded CSX described in the architecture document — with full access to the shared variable context, the discovered service endpoints, and any referenced customer libraries. It is used when declarative steps cannot express the required logic: conditional branching, iteration, a non-trivial transformation of captured data, or an assertion that depends on a proprietary object model. The script reads and writes the same variables that declarative steps use, so it composes seamlessly with the rest of the file. The step type is always written as the dotted type: script.csharp.

- id: derive-and-check
  type: script.csharp
  description: "Compute an expected checksum and assert it"
  code: |
    var userId = Vars.GetString("newUserId");
    var acct   = await Db("orders-db")
                       .QuerySingleAsync(userId);
    Assert.Equal("standard", acct.Plan);
    Vars.Set("checksum", Checksum.Of(acct));   // customer library

As the body grows, an author can move it out of the YAML and into a sibling .csx file, referenced by file instead of code:

- id: derive-and-check
  type: script.csharp
  description: "Compute an expected checksum and assert it"
  file: scripts/derive-and-check.csx

file and code are mutually exclusive — a step declares exactly one. The path is resolved relative to the .e2e.yaml file's own directory, and the referenced file's content is read once at compile time and spliced into the compiled submission exactly as an inline code body would be — same trust boundary, same lack of sandboxing, same absence of placeholder/secret substitution (§6.2). A file that does not exist is a validation error, reported before topology start (an authoring mistake, not a product defect — §12.1 Inconclusive).

Guidance

Reach for a script step only when the declarative types genuinely cannot express the intent. Declarative steps are validated by the schema, are readable by non-engineers, and are the unit the agentic Generator produces. A file that is mostly script steps has lost most of the DSL's benefits. Prefer file once a script body is long enough to want its own C#/OmniSharp editing support, or is shared across scenarios.

Security caution: Secret handling in script.csharp

Because script.csharp runs author-trusted code in the shared variable context, it can read any staged value — including connection strings and secret values accessed via Vars.Secrets.Resolve(...).Reveal(). The engine's redaction cannot guarantee that a value an author explicitly writes into a captured variable, a returned observation, or an exception message stays out of the report. Authors must not surface secrets from script bodies; prefer keeping credentials in ${secret:…} references consumed by typed provider fields. This applies identically whether the body is inline (code) or referenced (file).

5.7 Step families and the provider extension model

The sections above introduced each family with the provider used in its example, and noted in passing that other providers exist. This subsection collects the model into one place from the DSL author's perspective. The engineering view of the same model — interfaces, registry, compilation lifecycle — lives in section 13 of the Technical Architecture & Engineering Blueprint.

Every step type in the language is the combination of a family and a provider. The family names the abstract operation; the provider names the concrete technology that performs it. The author always writes the dotted form — type: db-assert.postgres, type: mq-publish.rabbitmq — and the engine resolves it to a registered implementation; the editor renders the same canonical dotted form in suggestions and errors.

The community catalogue at platform launch is summarised in the table below. The Core column lists what ships under Apache 2.0 with the engine; the Community column lists planned providers maintained outside the engine repository (hosted in the community hub or externally), or already delivered.

Family	Core (ships with the engine)	Community (planned or listed)
http	http.rest, http.soap	http.graphql, http.long-polling
rpc	—	rpc.grpc, rpc.json-rpc (available — the hub's first community provider), rpc.thrift
mq-publish	mq-publish.kafka, mq-publish.rabbitmq, mq-publish.nats, mq-publish.azureservicebus, mq-publish.redis	mq-publish.mqtt, mq-publish.sqs, mq-publish.eventhubs, mq-publish.pulsar, mq-publish.activemq-artemis, mq-publish.gcp-pubsub, mq-publish.azurestoragequeue, mq-publish.sns
mq-expect	mq-expect.kafka, mq-expect.rabbitmq, mq-expect.nats, mq-expect.azureservicebus, mq-expect.redis	mq-expect.mqtt, mq-expect.sqs, mq-expect.eventhubs, mq-expect.pulsar, mq-expect.activemq-artemis, mq-expect.gcp-pubsub, mq-expect.azurestoragequeue
db-assert	db-assert.postgres, db-assert.sqlserver, db-assert.mysql, db-assert.mongodb, db-assert.dynamodb	db-assert.oracle, db-assert.cassandra, db-assert.scylladb, db-assert.cockroachdb, db-assert.clickhouse, db-assert.cosmosdb, db-assert.neo4j, db-assert.mariadb, db-assert.ldap, db-assert.qdrant
cache-assert	cache-assert.redis, cache-assert.elasticsearch	cache-assert.opensearch, cache-assert.valkey, cache-assert.memcached
mail-expect	mail-expect.smtp	—
webhook-listen	webhook-listen.http	—
metrics-assert	metrics-assert.prometheus	—
storage-assert	storage-assert.s3	storage-assert.azblob, storage-assert.sftp, storage-assert.gcs
script	script.csharp	—
trace-expect	trace-expect.otlp	trace-expect.jaeger
realtime-expect (reserved)	—	realtime-expect.websocket, realtime-expect.sse, realtime-expect.signalr, realtime-expect.graphql-ws

Table 5.1 — The community catalogue at platform launch. The list is indicative; the authoritative list is the unified JSON Schema served by the installed engine. The dotted family.provider form (e.g., type: mq-publish.kafka) is the only accepted form for every family, regardless of how many providers it has registered. Placement of a provider in the Community column is a plan, not a promise of ordering or timing; the Vouched badge may be requested post-listing under the governance model (see GOVERNANCE.md).

Families marked (reserved) are name-reservations: the family name and intent are fixed by the platform team, and the full step-field specification is published when the family's first provider lands (trace-expect has now graduated out of this state, with trace-expect.otlp shipping as its Core provider). A provider whose steps observe a managed dependency (a store or broker declared under environment.dependencies) also requires engine support for that dependency type; the supported dependency types are listed in section 3, and additions ride the engine's release cadence. Providers that speak a protocol directly to a service under test (the http, rpc, realtime-expect, metrics-assert and trace-expect shapes) have no such prerequisite and can be delivered by the community self-contained — metrics-assert.prometheus and trace-expect.otlp are the shipped proof: the former scrapes a service declared under environment.services exactly like http.rest, and the latter's receiver is an in-process host resource exactly like webhook-listen.http's listener — neither adds a new dependency kind.

5.7.1 How adding a provider feels to a contributor

A contributor adding a new provider — say db-assert.scylladb — submits it to the community provider hub (vouchfx-providers). They either contribute their provider and test projects as source under the hub's community/ directory (a project folder plus a sibling .Tests folder, with a README.md), or list an externally hosted NuGet package in the hub's registry. In either case, they implement a small set of C# interfaces, write a JSON Schema fragment, add worked examples, and open a pull request. Once the basic requirements are met (hygiene gates: dependencies, documentation, examples), the provider (or, for externally hosted providers, its registry entry) merges into the hub and is listed in the registry. The DSL itself does not change; what changes is the unified schema served to the editor and the set of registered providers in the engine. From an author's perspective, db-assert.scylladb simply starts appearing in autocomplete the next time they update their engine.

The Vouched badge is optional and requested post-listing via a dedicated issue template in the hub. The badge requires a published checklist (VOUCHED_CHECKLIST.md) covering six items: multi-version conformance testing, documentation with use cases and limitations, security sign-off, Apache-2.0 licensing with DCO sign-off, MinEngineVersion declaration, and CsxFragment composition review. The engineering details of the contract — the C# interfaces a provider implements, the CsxFragment composition rules every provider's emitted code must respect, the engine-side compatibility commitment that protects providers across engine releases — are specified in Section 13.3 of the Technical Architecture & Engineering Blueprint. The hub's comprehensive implementation guide walks the complete end-to-end workflow, using the first community provider (rpc.json-rpc) as its reference implementation.

5.7.2 Choosing between providers in the same family

Two patterns make the choice explicit and reviewable. First, the provider is named in every step — reading the file makes it immediately clear that a test is running against Postgres rather than Oracle. Second, the file's environment.dependencies section declares the technology of every store and broker, and the validator refuses a step whose provider does not match the dependency's declared type. A db-assert.postgres step pointed at a MongoDB dependency is therefore a static error caught before any container starts, not a runtime exception buried in a test log.

The dotted form is the only accepted form

A bare family name (e.g., type: http, type: script) is not a valid step type, even for a family that currently has exactly one registered provider. This was changed pre-v1.0 (while the schema was still unpublished) precisely so that a family gaining a second provider can never silently change — or break — the meaning of a step type an existing test file already uses. Every step always names its provider explicitly: type: http.rest, type: script.csharp, type: db-assert.postgres, and so on. If an author mistypes a family name as a bare word, the validation error names the registered providers of the matching family (e.g. mistyping type: cache-assert reports that cache-assert.redis and cache-assert.elasticsearch are the registered options), so the fix is immediate.

cache-assert.elasticsearch — step fields

Asserts the state of an Elasticsearch index by submitting a Query DSL request and verifying the result set.

Field	Required	Description
target	Yes	The logical name of the elasticsearch dependency declared in environment.dependencies.
index	Yes	The Elasticsearch index name to query. Supports {placeholder} substitution.
query	No	A JSON string containing an Elasticsearch Query DSL body (e.g. {"query":{"term":{"id":"{orderId}"}}}). Supports {placeholder} substitution in field values. Defaults to a match-all query when omitted.
expect.count	No	Exact number of hits required. Mutually exclusive with expect.min-count.
expect.min-count	No	Minimum number of hits required (default 1 when neither count assertion is specified).
expect.fields	No	List of {field, value} pairs asserted against the _source of the first hit. Each entry must match exactly.

Use verifyMode: RETRY with a timeout when the index may lag behind the write path (eventual consistency). {placeholder} references in query are resolved at execution time after all preceding capture expressions have been applied.

5.8 The metrics-assert family

A metrics-assert step scrapes a Prometheus text-exposition endpoint — normally the system under test's own /metrics — and asserts on one metric's numeric value, optionally scoped by a label subset. This is the family that proves a business transaction actually moved a counter or gauge the SUT exposes: a 2xx HTTP response tells you the SUT accepted a request, but only the metric tells you it did the work. metrics-assert is a brand-new family; its Core provider is metrics-assert.prometheus. Like every family, the bare type: metrics-assert form is not accepted — the dotted form is always required.

Unlike mq-publish/mq-expect/cache-assert, this family adds no new infrastructure kind: it scrapes a service declared under environment.services — exactly the mechanism http.rest already uses — rather than a managed dependency under environment.dependencies. A community contributor can therefore ship a metrics-assert provider self-contained, with no engine-side dependency-type work (§5.7's table already lists this family among the self-contained shapes, alongside http, rpc, realtime-expect and trace-expect).

Ambiguous-selection policy. Exactly one sample must match the declared metric name plus every declared labels entry (a label subset match — the real sample may carry additional labels the author did not mention). Zero matches is a Fail ("found":false); more than one match is also a Fail ("ambiguous":true), never a silently-resolved "first match wins" pick — an under-specified label set is an authoring error the author must fix, not a decision the engine makes on their behalf.

The delta idiom. Because the DSL's capture:/{placeholder} substitution never evaluates arithmetic, proving a transaction incremented a counter by exactly N needs an intermediate script.csharp step to compute the expected post-transaction value. The directly-expressible, still-useful form is capture-before / assert-after with a non-decrease bound: capture the metric's value before the transaction, trigger the transaction, then assert min against the captured baseline (see the worked example below). metrics-assert.prometheus exposes the matched value to capture: at JSONPath $.value, evaluated against a small synthesised observation object {"metric":<name>,"value":<n>,"labels":{...}} — the very same object the Pass-verdict observation carries.

verifyMode: RETRY is this family's canonical mode. A metric update frequently lags the HTTP response that triggered it (the same post-stimulus convergence gap mq-expect and db-assert steps already accommodate), so most metrics-assert steps are written with verifyMode: RETRY and a timeout.

metrics-assert.prometheus — step fields

Field	Required	Meaning
target	Yes	Logical name of the service to scrape, as declared under environment.services — normally the system under test.
path	No	The scrape path. May contain {placeholder} and ${secret:...} tokens. Defaults to /metrics when omitted.
metric	Yes	The Prometheus sample (metric) name to select, e.g. orders_processed_total. May contain {placeholder} and ${secret:...} tokens.
labels	No	Map of required label name to expected value. A sample matches only when it carries every declared label with exactly the expected value (a subset match). Values may contain {placeholder} and ${secret:...} tokens.
expect.value	No	Expected exact value, compared with a 1e-9 relative tolerance. A decimal string; may contain {placeholder} / ${secret:...} tokens.
expect.min	No	Inclusive lower bound for the matched sample's value.
expect.max	No	Inclusive upper bound for the matched sample's value.

At least one of expect.value, expect.min, or expect.max must be declared; all three may combine. A non-200 scrape response is a Fail (the endpoint answered — it just did not serve metrics), never an EnvironmentError; a connection failure or timeout maps to EnvironmentError / Inconclusive exactly as http.rest does (§12.1 of the architecture blueprint).

Example (the delta idiom: capture-before, trigger, assert-after):

- id: capture-baseline
  type: metrics-assert.prometheus
  target: app
  metric: orders_processed_total
  labels:
    status: success
  expect:
    min: "0"
  capture:
    ordersBefore: $.value

- id: trigger-order
  type: http.rest
  target: app
  method: POST
  path: /orders
  body: '{"customerId":"cust-1","amount":49.99}'
  expect:
    status: 202

- id: assert-orders-processed
  type: metrics-assert.prometheus
  target: app
  metric: orders_processed_total
  labels:
    status: success
  expect:
    min: "{ordersBefore}"
  verifyMode: RETRY
  timeout: 15s

5.9 The storage-assert family

A storage-assert step observes an object the system under test wrote to an S3-compatible object store and asserts on its existence, size, content type, user metadata, and (optionally) the body's SHA-256 digest or a body substring. This is the family that proves a business transaction actually wrote a file — an export, a generated report, an uploaded attachment — the same "did the system actually do the thing" signal metrics-assert provides for counters/gauges and db-assert provides for rows/items. storage-assert is a brand-new family; its Core provider is storage-assert.s3. Like every family, the bare type: storage-assert form is not accepted — the dotted form is always required.

Unlike metrics-assert (which adds no new infrastructure kind), storage-assert.s3 does require a new managed-dependency type: minio, a MinIO container the engine provisions for the declared dependency (§3.2's minio row). MinIO speaks the S3 API, so the same provider works unchanged against any S3-compatible endpoint reachable at a ServiceURL.

The exists coherence rule. expect.exists: false excludes every content expectation (size, minSize, sha256, contentContains, contentType, metadata) — an absent object has no content to assert on. expect.size and expect.minSize are mutually exclusive: declare an exact size or a lower bound, not both.

The bounded-GET rule. Every attempt HEADs the object first (existence, size, content type, metadata — no body transfer). Only when sha256 or contentContains is declared does the step GET the body, and only after every HEAD-based check has already passed; a body larger than 16 MiB is refused with a Fail noting the cap, never fetched. A contentContains mismatch reports only a boolean match flag and the body length — the searched substring and the body itself are never echoed into the observation (the same "no content in observations" discipline webhook-listen.http/mail-expect.smtp apply to captured request/message bodies). SHA-256 digests, by contrast, are safe to report verbatim: a one-way, fixed-length hash cannot leak the underlying content.

verifyMode: RETRY is this family's canonical mode — the eventually-consumed idiom. A file write frequently lags the HTTP response that triggered it (the same post-stimulus convergence gap mq-expect/db-assert/metrics-assert steps already accommodate), so most storage-assert steps are written with verifyMode: RETRY and a timeout; the HEAD call is read-only and idempotent, so every poll attempt is safe to repeat.

Capture support. The Pass observation exposes etag, versionId, size, and exists to capture: via JSONPath ($.etag, $.versionId, $.size, $.exists) — mirrors metrics-assert.prometheus's synthesised-observation capture mechanism exactly.

storage-assert.s3 — step fields

Field	Required	Meaning
target	Yes	Logical name of the minio dependency, as declared under environment.dependencies.
bucket	Yes	The S3 bucket name. May contain {placeholder} and ${secret:...} tokens.
key	Yes	The S3 object key. May contain {placeholder} and ${secret:...} tokens.
expect.exists	No	Whether the object is expected to exist. Defaults to true when omitted.
expect.size	No	Exact expected object size in bytes. Mutually exclusive with expect.minSize.
expect.minSize	No	Inclusive minimum expected object size in bytes. Mutually exclusive with expect.size.
expect.sha256	No	Expected lowercase hex SHA-256 digest of the object body. Triggers the bounded GET (16 MiB cap).
expect.contentContains	No	A substring the object body (decoded as UTF-8) must contain. Triggers the bounded GET; never echoed into the observation.
expect.contentType	No	Expected Content-Type, compared against the HEAD response.
expect.metadata	No	Map of user-metadata key to expected value, compared against the HEAD response's object metadata.

Example (the eventually-consumed idiom: trigger, then RETRY until the export lands):

- id: trigger-export
  type: http.rest
  target: app
  method: POST
  path: /reports/export
  body: '{"format":"csv"}'
  expect:
    status: 202

- id: assert-report-written
  type: storage-assert.s3
  target: reports
  bucket: reports
  key: "exports/daily-report.csv"
  verifyMode: RETRY
  timeout: 15s
  expect:
    exists: true
    minSize: "1"
    contentType: "text/csv"
  capture:
    reportEtag: $.etag

5.10 The trace-expect family

A trace-expect step scans the spans exported over OTLP/HTTP by the system under test's own OpenTelemetry SDK and asserts that one matches a declared shape. This is the platform's flagship distributed assertion: it proves the causal chain of a business transaction — that a request into one service genuinely produced downstream work, correlated by a single trace — a proof no single-service assertion (an HTTP 2xx, a database row, a queue message) can give on its own. trace-expect is a brand-new family; its Core provider is trace-expect.otlp. Like every family, the bare type: trace-expect form is not accepted — the dotted form is always required.

How it works. The engine stands up an ephemeral, host-owned OTLP/HTTP receiver for each declared receiver name and stages its base URL at svc::<receiver> and the plain Vars[<receiver>] key — exactly the webhook-listen.http host-resource pattern (§5.5). In principle the system under test's OpenTelemetry SDK is pointed at it via the standard OTEL_EXPORTER_OTLP_ENDPOINT environment variable (with OTEL_EXPORTER_OTLP_PROTOCOL=http/json — see the JSON-only limitation below), so the SDK's own export path does the rest with no test-only shim in the SUT.

Known limitation (v1): host resources cannot yet be wired into environment.services[].env. The receiver is a per-scenario host resource started only once the topology (and therefore the SUT container) is already running (§5.5's env: support — §3.2.4 — resolves only ${conn:<dependency>} references at container-configure time, before the receiver exists). A containerised SUT therefore cannot yet be pointed at the per-scenario receiver via its own startup configuration; it would need to target a standing OTel collector whose address is stable for the whole suite run, which then forwards to wherever the engine happens to be listening — outside this family's v1 scope. Until that gap closes, the honest, self-contained way to exercise trace-expect.otlp is to SIMULATE the export with a script.csharp step that posts a synthetic OTLP/HTTP JSON payload directly to the receiver's staged URL (Vars["<receiver>"]), exactly as the worked example below does — the same "HONEST disclosure" pattern the engine's own webhook capstone test uses for webhook-listen.http. This still exercises the full receiver + staging + RETRY + assertion path end-to-end; only the "real instrumented SUT container" half is simulated.

The traceparent idiom. An earlier http.rest step injects a W3C traceparent header on its outbound request (a suite typically threads a fresh one forward via capture/{placeholder}, or generates one with a script.csharp step). The SUT's auto-instrumented OpenTelemetry SDK continues that same trace, so the trace-expect.otlp step's traceId criterion can simply reuse the identical traceparent value — the provider extracts the 32-hex trace-id segment from a full traceparent automatically.

verifyMode: RETRY is this family's canonical mode — more so than any other family. A real OpenTelemetry SDK batches and exports spans on a periodic timer (commonly around five seconds), so the very next step will almost always run before the export has landed. Every trace-expect.otlp step should be written with verifyMode: RETRY and a timeout generous enough to span at least one export cycle.

The honest caveat. trace-expect.otlp proves that a trace with the expected shape was emitted — it is telemetry, not a state assertion. It cannot, on its own, prove the transaction's business effect (that a row was written, a message was published, a counter moved). Pair it with a db-assert, mq-expect, metrics-assert, or storage-assert step for that proof; use trace-expect.otlp for the causal-chain proof those families cannot give.

trace-expect.otlp — step fields

Field	Required	Meaning
receiver	Yes	Logical name of the host-owned OTLP/HTTP receiver whose captured spans this step asserts against. The engine stands the receiver up and stages its base URL at svc::<receiver> (and the plain Vars[<receiver>] key), ready to be handed to the SUT's OTel SDK configuration, e.g. OTEL_EXPORTER_OTLP_ENDPOINT: "{traces}".
match.traceId	Yes	Required expected trace id — the transaction this assertion is tied to. Accepts either a bare 32-hex trace id or a full W3C traceparent value (00-<32hex>-<16hex>-<2hex>) — the 32-hex segment is extracted automatically. Matched case-insensitively. May contain {placeholder} / ${secret:...} tokens. Never omit it: a match with no trace id would accept any span with the right service/spanName/attributes, from any trace, ever exported (see the security note below) — validation rejects a match with no traceId.
match.service	No	Expected exporting-resource service name (the span's resource-level service.name attribute). Matched by ordinal equality. An optional refinement on top of traceId, never a substitute for it.
match.spanName	No	Expected span name (operation name). Matched by ordinal equality. An optional refinement on top of traceId.
match.attributes	No	Map of expected span-attribute key to expected string value (a subset match — the real span may carry additional attributes not mentioned here). An optional refinement on top of traceId.

All declared criteria are conjunctive (every one must hold for a span to match).

OTLP encoding: JSON only in v1. The receiver accepts OTLP/HTTP's JSON encoding (Content-Type: application/json) only — the far more common protobuf wire encoding (application/x-protobuf) is rejected 415 with a message naming the limitation. Configure your OTel exporter with OTEL_EXPORTER_OTLP_PROTOCOL=http/json; every major OpenTelemetry SDK supports this.

Capture. The matched span's fields (traceId, spanId, parentSpanId, name, serviceName, statusCode, and attributes.<key>) are exposed to capture: via simple dotted-path expressions ($.spanId, $.attributes.orderId) — mirroring the synthesised-observation idiom metrics-assert.prometheus/storage-assert.s3 already use, though only simple field access is supported (no array indexing, wildcards, or recursive descent).

No forged-match risk despite no listener-token gate. Unlike webhook-listen.http's receiver, the OTLP receiver does not add an unguessable path-token segment to its URL — because the standard OTEL_EXPORTER_OTLP_ENDPOINT convention must remain a plain, unmodified base URL for a real OpenTelemetry SDK to work against it unmodified. This is safe because match.traceId is required and enforced (rejected at validation time if omitted, not merely a convention an author could accidentally skip) — a trace-expect.otlp assertion always requires a specific trace id drawn from the current scenario's own transaction, so an off-network onlooker who cannot observe that id cannot manufacture a matching span.

Honest caveat on the ring buffer: an availability trade-off, not a forgery risk. The receiver's ring buffer (§ above, 10,000 spans) bounds retained spans, not accepted ones: a sustained flood of forged or unrelated exports can still evict the real span from the retained window before a step scans it. This is deliberately not a false Pass — it surfaces as a loud Fail with a non-zero evicted count in the observation (the difference between the total spans the receiver has ever received and the number currently retained), distinguishing "the buffer is saturated" from "the SUT genuinely never exported the span". The ephemeral-port/per-scenario-lifetime and bounded-body/bounded-buffer mitigations keep the exploit window and a flood's worst-case cost small; they do not claim to make a sustained-flood denial impossible.

Example (the traceparent idiom; the OTLP export is SIMULATED per the known limitation above — see examples/trace-expect-otlp.e2e.yaml for the fully worked, runnable version):

environment:
  services:
    app:
      image: myorg/orders-api:latest
  dependencies:
    orders-db:
      type: postgres

variables:
  traceparent: "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"

steps:
  - id: trigger-order
    type: http.rest
    target: app
    method: POST
    path: /orders
    headers:
      traceparent: "{traceparent}"
    body: '{"customerId":"cust-1","amount":49.99}'
    expect:
      status: 200

  - id: simulate-otlp-export
    type: script.csharp
    description: >
      SIMULATES the SUT's OpenTelemetry SDK exporting the span it recorded for the request
      above (see the known limitation above). A real, instrumented SUT would export this
      itself; this step exists only because environment.services[].env cannot yet reference
      a per-scenario host resource's URL.
    code: |
      var receiverUrl = (string)Vars["traces"];
      var payload = "{\"resourceSpans\":[{\"resource\":{\"attributes\":[{\"key\":\"service.name\",\"value\":{\"stringValue\":\"app\"}}]},\"scopeSpans\":[{\"spans\":[{\"traceId\":\"4bf92f3577b34da6a3ce929d0e0e4736\",\"spanId\":\"00f067aa0ba902b7\",\"parentSpanId\":\"\",\"name\":\"POST /orders\"}]}]}]}";
      var client = new System.Net.Http.HttpClient();
      try {
        var content = new System.Net.Http.StringContent(payload, System.Text.Encoding.UTF8, "application/json");
        await client.PostAsync(receiverUrl + "/v1/traces", content);
      } finally { client.Dispose(); }

  - id: assert-order-span
    type: trace-expect.otlp
    receiver: traces
    verifyMode: RETRY
    timeout: 30s
    match:
      traceId: "{traceparent}"
      service: app
      spanName: "POST /orders"
    capture:
      spanId: "$.spanId"

  - id: assert-order-persisted
    type: db-assert.postgres
    description: >
      The state assertion trace-expect.otlp's honest caveat calls for: the trace proves the
      transaction was CAUSALLY handled; this proves it had the expected EFFECT.
    target: orders-db
    verifyMode: RETRY
    timeout: 15s
    query: SELECT status FROM orders WHERE customer_id = 'cust-1'
    expect:
      rowCount: 1
      row:
        status: "created"

6. Variable Capture and Substitution

A meaningful end-to-end test is stateful: a value produced by one step is consumed by another. This section defines the two halves of that mechanism — how a value is captured out of a step's result, and how a captured value is substituted back into a later step.

6.1 Capturing values

The capture field, available on any step, is a map from a variable name to a capture expression. After the step completes, each expression is evaluated against the step's result and the value is written into the shared context under the given name. An author may specify a capture entry in two forms:

• Bare scalar (JSONPath, backward-compatible): id: "$.id" — the expression defaults to JSONPath and is evaluated against the step result as JSON. A bare scalar is the recommended concise form.
• Explicit single-key mapping (JSONPath or XPath): id: { jsonpath: "$.id" } or id: { xpath: "//order/id" } — the mapping's single key specifies the query language. Use the XPath form to extract values from XML response bodies — http.rest calls to XML-returning services, or http.soap calls, whose response is always XML (§5.1).

JSONPath uses JsonPath.Net (§5.7 libraries) and evaluates $.id to read a top-level field, $.payload.accountId to reach nested fields. XPath evaluates against the XML body; a non-XML response, invalid XPath expression, or no matching element results in a capture miss, which counts as an inconclusive outcome (timeout/unmet precondition) rather than a test failure.

Examples:

capture:
  newUserId: "$.id"                         # Bare scalar → JSONPath (back-compat)
  customerId: { jsonpath: "$.customer.id" } # Explicit JSONPath
  orderId: { xpath: "//order/@id" }        # XPath into XML result

6.2 Substituting values

Anywhere a value is expected in a declarative step, a captured or seeded variable can be substituted with brace syntax: {newUserId} is replaced, at execution time, by the current value of the newUserId variable. Substitution also works with secret references: {foo} reads from the context, whilst ${secret:source/path} resolves from a configured secret source. Both are resolved at step-execution time (never at compile time), ensuring secrets are never baked into the compiled code. Substitution works inside:

• HTTP paths, headers, and request body (the body is treated as a template string if a scalar, or serialised to JSON and then used as a template if a YAML mapping or sequence).
• Kafka topic names, message keys, plain payloads, Avro field values, and message headers.
• Database query strings and parameter values.
• Webhook listener URLs (via the listener name staged at {<listener>}), and all fields of the match criteria.

Because substitution happens at execution time, a placeholder always reflects the most recent value — which is what lets a generated identifier from step one appear in the assertions of step five.

script.csharp does not participate in placeholder substitution or secret redaction. Unlike every other Core step family, a script.csharp step's code (or, equivalently, the content of the file it references via file) is Turing-complete C# spliced into the compiled submission verbatim — the engine performs no {placeholder} substitution and no ${secret:source/path} resolution on it, and SecretString redaction is not applied. This also means the file path itself is not substituted — a {placeholder} written inside file: is treated as a literal string, not resolved, and will simply fail to resolve to a real path. The author is trusted (it is their own test code) and may read any value in Vars directly. To use a secret from script.csharp, resolve it explicitly via the secret accessor in code rather than expecting template substitution.

6.3 A worked thread of state

The three steps below show the full lifecycle of a single value. The reference scenario creates a user, then follows the generated identifier through a Kafka event and into a database projection.

Step	Writes to context	Reads from context
create-user (http)	newUserId, captured from $.id of the response.	tenantId, basePath (seeded constants).
expect-billing-event (mq-expect)	billingAccountId, captured from the message payload.	newUserId, matched against the message key and payload.
assert-mongo-projection (db-assert)	—	newUserId and billingAccountId, both asserted against the stored document.

Table 6.1 — One identifier threaded across three heterogeneous steps via the shared context.

6.4 Secret references

Some values a test needs must never be written into the file: a database password, an API key, an OAuth client secret, a webhook signing key. These are expressed as ${secret:source/path} references, which look like substitutions but resolve differently. A normal {variable} is replaced from the shared context the author populated; a ${secret:…} reference is resolved by the engine at runtime from a configured secret source, and its value lives only in memory for the duration of the run. The reference is what appears in source control; the literal value never does.

environment:
  dependencies:
    orders-db:
      type: postgres
      connectionString: "${secret:env/ORDERS_DB_CONN}"

steps:
  - id: call-protected-endpoint
    type: http.rest
    target: orders-api
    method: GET
    path: "{basePath}/admin/report"
    headers:
      Authorization: "Bearer ${secret:vault/secrets/api-keys#admin-token}"

The source prefix — env for an environment variable, vault for HashiCorp Vault KV v2, file for a git-ignored local file (deferred), and cloud for Azure Key Vault or AWS Secrets Manager (deferred) — is chosen by runner configuration, not by the file, so the same test resolves from an environment variable locally and from Vault in CI without any edit.

The vault source addresses a HashiCorp Vault KV v2 store with the syntax ${secret:vault/<kvPath>#<field>}, where <kvPath> is the logical KV path (e.g. secrets/api-keys) and <field> is the key within that path's data object to return (e.g. admin-token). The KV path and field are both mandatory. Configuration is via three optional environment variables: VAULT_ADDR (the Vault server address, e.g. http://127.0.0.1:8200), VAULT_TOKEN (the access token), and VAULT_KV_MOUNT (the KV v2 mount name; defaults to secret). Resolution happens at step-execution time; the token and resolved value never leak into logs or reports.

The reporting layer redacts any value that originated from a secret reference, showing the reference placeholder in its place, so a secret cannot leak through a report or a captured-variable thread. Captured and substituted variables are traceable end-to-end in the terminal report's provenance section, with secret-derived values always shown redacted (reference only, never the literal value). The resolution mechanism and its tier mapping are specified in Section 17 of the companion Technical Architecture & Engineering Blueprint.

7. Verification Modes and Asynchronous Assertions

Asynchrony is the defining difficulty of distributed testing. A REST call may return instantly, but the Kafka event it triggers, and the database row that event produces, arrive some unpredictable moment later. The verification mode is the language construct that handles this without forcing the author to reason about timing.

7.1 IMMEDIATE versus RETRY

Every step runs in one of two verification modes. IMMEDIATE, the default, evaluates the step's assertions exactly once: it suits a synchronous HTTP response, where the answer is available the moment the call returns. RETRY tells the engine that the asserted condition is expected to become true eventually rather than immediately. The engine then polls — re-running the underlying query or consume operation — until either the assertions hold or the step's timeout elapses.

7.2 What RETRY compiles to

A RETRY step does not compile to a loop with a fixed sleep. As the architecture document describes, the compiler emits a Polly v8 resilience policy that performs bounded exponential-backoff polling. The interval between attempts grows — short at first, then longer — so a condition that becomes true quickly is detected quickly, while a slow condition does not generate excessive load. The author never sees this machinery; they write verifyMode: RETRY and a timeout, and the engine guarantees the same behaviour on a fast laptop and a loaded CI runner.

The engine polls the step's underlying assertion (re-running its query or consume operation) and records each attempt as a separate step-attempt event in the verdict stream, including the attempt index and timing. This makes the polling timeline visible in reporting without requiring the suite to re-run. When an assertion passes, the step succeeds immediately. When the step's timeout elapses without the assertion passing, the verdict is Inconclusive (not Fail — see the verdict taxonomy in §12.1), because the timeout reflects only the bounded polling window, not a defect in the system under test. An environment error encountered during an attempt — such as an unreachable broker — is terminal and not retried; it writes an EnvironmentError verdict for the step.

Why this matters

Hand-rolled waits — a fixed sleep, or a while loop — are the single largest source of flaky integration tests, because the right delay differs on every machine. Making RETRY a first-class language construct removes the temptation to write them and is the practical expression of the design objective “determinism by default”.

7.3 Tuning the polling window and backoff

The timeout field sets the outer bound for the polling window — the longest the engine will keep trying before resolving the step's verdict. Within that window, the engine applies exponential backoff with these defaults: initial delay of 200 milliseconds, doubling on each attempt with random jitter, and a maximum interval of 5 seconds. These defaults are chosen to be conservative — quick on fast systems, gentle on slow ones — and do not require author configuration. The window also bounds an in-flight attempt: where a provider's client supports cooperative cancellation, an attempt still running when the window elapses is cut at the boundary rather than running on to its transport bound; each attempt otherwise keeps the provider's transport timeout as its own per-attempt limit.

An optional pollInterval field is reserved in the schema as documentation of intent, but it is not yet parsed or honoured by the execution engine; the timeout is the real bound. A future release may expose pollInterval to allow authors to tune the base delay or backoff strategy. For now, authors should rely on the default backoff and set a timeout that reflects a realistic worst case for the system under test rather than an arbitrarily large value, because an over-long timeout turns a genuine failure into a slow failure.

8. The JSON Schema Contract

The entire language is governed by a single JSON Schema. This section explains the role that schema plays, how it is structured, and why having exactly one of it is an architectural decision rather than a convenience.

8.1 One schema, three consumers

The schema is consumed by three independent parts of the platform, and the fact that they share one source is what keeps them honest.

Consumer	How it uses the schema
The compiler	Validates a test file before any container starts. A structurally invalid file fails fast, with an error that points at the offending line, rather than failing obscurely mid-execution.
The VSCode extension	Drives autocomplete and inline validation. Because the editor reads the same schema the compiler enforces, it can never suggest a field the compiler would reject.
The documentation generator	Produces the reference documentation for the language directly from the schema, so the documentation cannot drift out of step with the implementation.

8.2 Structure of the schema and how it is assembled

The schema is organised so that the editor can give precise, context-aware suggestions. The root object defines the four top-level sections. The steps array uses a discriminated union keyed on the type field: once the author writes type: http.rest, the schema constrains the remaining fields to those valid for that step kind, which is what lets the editor autocomplete method and expect but not topic. Shared definitions — the common step fields, the assertion block, the duration format — are factored into reusable schema fragments referenced from each step kind.

Crucially, the unified schema is assembled rather than authored. The platform's compiler walks the registered providers at startup, collects each provider's JSON Schema fragment, and merges those fragments into the discriminated union. This is how a new provider becomes visible to the editor without any change to the editor itself: the contributor adds the provider, its fragment joins the union, and the next time the editor fetches the schema the new step kind appears in autocomplete with its provider-specific field set. The mechanism is described in detail in the Technical Architecture & Engineering Blueprint; the relevant point for DSL authors is that the schema they validate against is always exactly the schema the installed engine will execute.

Each provider's schema fragment contributes the fields specific to that step type. For example, the mq-publish.kafka provider contributes fields for target, topic, key, payload, headers, and the optional avro block; the mq-expect.kafka provider contributes target, topic, match, and the optional avro block. Dependencies in the environment section may also declare provider-specific flags — for example, a kafka dependency's schemaRegistry: true flag is a Kafka-specific extra that signals the provisioning of a Confluent Schema Registry sidecar.

The v1 schema is frozen

The composed v1 JSON Schema — the root language schema plus the fragments from all twenty-five Core providers — is frozen. The schema is versioned and self-identifies via the x-vouchfx-schema-version: v1 annotation; the composed artifact is snapshotted in the test suite and validated byte-for-byte on every build. Any change to the schema (a new provider fragment, a tightened enum, a new keyword) requires deliberate regeneration and review of the frozen golden. This freeze ensures authors building against the v1 schema can rely on a stable contract for their test files.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "E2E Integration Test",
  "type": "object",
  "required": ["steps"],
  "properties": {
    "metadata":    { "$ref": "#/$defs/metadata" },
    "environment": { "$ref": "#/$defs/environment" },
    "variables":   { "type": "object" },
    "steps": {
      "type": "array",
      "items": { "$ref": "#/$defs/step" }
    }
  },
  "$defs": {
    "step": {
      "oneOf": [
        { "$ref": "providers/http.rest.json"           },
        { "$ref": "providers/mq-publish.kafka.json"    },
        { "$ref": "providers/mq-expect.kafka.json"     },
        { "$ref": "providers/db-assert.postgres.json"  },
        { "$ref": "providers/webhook-listen.http.json" },
        { "$ref": "providers/script.csharp.json"       }
        /* additional provider fragments are appended here */
      ]
    }
  }
}

8.3 Schema versioning across two axes

Because the unified schema is composed at startup from installed provider fragments, what an author validates against is a combination of two independently versioned things: the engine's stable language schema, and the set of provider fragments currently installed. The versioning model has to reflect that, or a file that validates today can mysteriously fail tomorrow when an unrelated provider is upgraded.

The platform addresses this with two version axes that compose explicitly. The first is the language schema version — the engine's contract for the four top-level sections, the common step fields, and the generic rules every provider's fragment must comply with. This version changes only when the engine changes those contracts and follows a deprecation window. A file may declare the language version it targets through a schemaVersion: v1 field in its metadata block.

The second is the set of provider versions currently registered, which any given suite's reproducibility envelope records (see Section 14.7 of the architecture blueprint). The engine treats minor and patch provider upgrades as backward-compatible additions to the unified schema; a major provider version may change the fragment in incompatible ways and is gated on the provider declaring a compatible MinEngineVersion. The combined effect: a file declaring schemaVersion: v1 will validate against any v1-compatible engine, and the reproducibility envelope of every run identifies exactly which provider versions were active so a divergence is always explainable rather than mysterious.

The VSCode extension maps each file to the correct language schema version, fetches the locally-installed provider fragments, and composes them as the engine does. An author working in an older repository against a newer engine still receives validation that matches the language their files were written for.

9. The VSCode Extension: Overview

A custom DSL imposes a real cost on its users: unfamiliar syntax, features that cannot be discovered, and references that break silently when a service changes. The VSCode extension exists to pay that cost down. Its goal is to make authoring the YAML feel like authoring a well-supported mainstream language — with autocomplete, inline error reporting, and semantic checks that understand the project around the test file. The remaining sections specify how it achieves that.

9.1 What the extension provides

The extension delivers four capabilities, each addressed in detail in a later section. Together they cover the full authoring loop, from typing a field name to catching a reference that no longer resolves.

Capability	What it does for the author	Specified in
Schema-driven YAML support	Real-time autocomplete and structural validation of the .e2e.yaml file against the JSON Schema.	Section 10
Embedded C# intelligence	Full C# IntelliSense, linting, and NuGet awareness inside script blocks.	Section 11
Semantic project validation	Cross-references the test against OpenAPI, AsyncAPI, and SQL schemas to catch references to things that do not exist.	Section 12
Run and debug integration	Executes a suite locally or against the cloud, surfaces results inline, and authenticates to an enterprise IdP when required.	Section 13

9.2 Architecture of the extension

The extension is not a monolith. It is best understood as a thin VSCode-facing shell that coordinates several language-intelligence providers. Static, schema-based features are handled through declarative contribution points in the extension manifest. Anything requiring real understanding of code or project structure is delegated to a language server communicating over the Language Server Protocol. This separation matters because it lets the heavyweight semantic analysis run out of process, keeping the editor responsive, and it lets the C# intelligence reuse the existing .NET tooling rather than reimplementing it.

10. Schema-Driven YAML Support

The first and most fundamental capability is making the YAML itself easy to write correctly. This is achieved declaratively, without any custom code, by binding the platform's JSON Schema to the test files through VSCode's contribution-point mechanism.

10.1 Contribution points

A VSCode extension declares its capabilities in the contributes section of its package.json manifest. Two contribution points carry the schema binding. The yamlValidation point — provided by the YAML language support that the extension depends on — associates a file pattern with a JSON Schema. The jsonValidation point does the same for any JSON artefacts the platform uses. By mapping the pattern *.e2e.yaml to the hosted schema, the extension gives every test file real-time IntelliSense and validation with no further work.

// package.json — contributes section
"contributes": {
  "yamlValidation": [
    {
      "fileMatch": "*.e2e.yaml",
      "url": "https://schemas.platform.example/e2e/v1.json"
    }
  ]
}

10.2 What the author experiences

With the binding in place, the editor behaves as it would for any well-supported configuration format. As the author types a field name, valid keys for the current context are offered — and because the schema uses a discriminated union on type, those suggestions are specific to the step type being written. A missing mandatory field is underlined immediately. An invalid value — a status code that is not a number, a verification mode that is not IMMEDIATE or RETRY — is flagged the moment it is typed. Hovering a field shows its description, drawn from the schema. None of this requires the author to consult external documentation, which is the design objective of discoverability made concrete.

10.3 Schema hosting and resolution

Delivered: The frozen v1 JSON Schema is bundled with the extension and bound declaratively via the yamlValidation contribution point, served by the Red Hat YAML language server dependency. The schema is a byte-for-byte copy of the engine's canonical v1 schema, and a CI test (VsCodeShippedSchemaSyncTests) enforces that the shipped copy remains in sync. For enterprise deployments, the vouchfx.schemaPath configuration setting allows pointing at an alternative copy of the same v1 schema (e.g., offline or internal hosting); this setting relocates the schema only and does not diverge from the frozen v1 contract.

10.4 Test Explorer integration

Delivered: The extension contributes a VSCode Test Controller that discovers *.e2e.yaml files in the workspace and renders a tree of scenarios and steps in the Test Explorer view. Running tests from the Test Explorer invokes the vouchfx CLI and surfaces per-step verdicts with line-level error decoration in the editor. The live end-to-end acceptance (verdicts reflected and Fail-decorated lines shown) is contingent on the companion CLI release (PR #148, which delivered the --events and --no-decorations flags); until that merge, the Test Explorer run handler fails soft with items marked errored and a vouchfx.cliPath configuration notice.

Discovery and tree structure

A FileSystemWatcher monitors the workspace for *.e2e.yaml files. Each discovered file becomes a parent TestItem, labelled by its metadata.name or filename if metadata is absent. Within each file, the extension parses the YAML outline and creates a child TestItem for every step bearing an id field, anchored at the 0-based line number on which the step declaration begins. This information comes from a pure parsing routine (parseE2eOutline) that uses the yaml package's LineCounter to map byte offsets to line numbers; malformed YAML files yield empty outlines rather than throwing. The tree updates automatically as files are created, modified, or deleted; an explicit refresh button (built into VSCode's Test Explorer) rescans the workspace.

Run profile and CLI contract

The extension registers a Run profile that, when invoked, spawns the vouchfx CLI for each selected file with the arguments:

<cliPath> run <file> --events <tmpEventsFile> --no-decorations

The <cliPath> is resolved from the vouchfx.cliPath configuration setting (default: bare "vouchfx" command); the <file> is the absolute path to the .e2e.yaml file; and <tmpEventsFile> is a unique temp-directory path where the CLI writes its JSON Lines event stream. The --no-decorations flag suppresses the engine's ANSI colour and formatting codes, as the Test Explorer surface handles visual styling. This CLI contract is the interface between the VSCode extension (consumer) and the reporting/CLI work (producer, delivered in PR #148). Until #148 merges, the two flags are unavailable; the run handler detects this and degrades gracefully (see Fail-soft error handling below).

Verdict mapping to editor state

The extension reads the emitted --events stream via a pure JSON Lines parser (parseEvents) that extracts scenario-started, step-completed, and scenario-completed events. Each step event carries the step's id and a verdict wire token: "PASS", "FAIL", "ENV_ERROR", or "INCONCLUSIVE". The pure mapVerdict function maps these tokens to VSCode TestItem states and human-readable labels:

Wire token	Editor state	Label	Meaning
PASS	passed	Pass	The step's assertions held.
FAIL	failed	Fail	The step's assertions did not hold (product defect).
ENV_ERROR	errored	Environment error	Infrastructure failure (container unhealthy, service unreachable, etc.); not a defect in the system under test.
INCONCLUSIVE	skipped	Inconclusive	The engine could not reach a verdict (timeout, unmet dependency on earlier step's capture); the result is uncertain rather than green or red.

This four-way mapping keeps the verdict taxonomy distinct on the editor surface, mirroring the §12.1 invariant that the four verdicts must never be conflated. Environment errors and inconclusive results are never silently reported as passes.

Failed step decoration

When a step completes with a FAIL verdict, the extension attaches a TestMessage to the TestItem whose location is the step's YAML range (start and end line). VSCode uses this location to decorate the originating line in the editor with an error gutter icon and inline message, so the author sees immediately which step needs attention without switching panels.

YAML-to-line mapping

The event stream emitted by the --events flag carries no source-location information (step events include only the step id, not the YAML line number). The extension's outline parser runs independently on the file text, building a map of step id → line range; when a step event arrives, it is matched by id to the corresponding outline entry and decorated at that line. This decoupling — discovery and run are separate pure functions — means the editor surface tolerates file changes between discovery and run; if the file changed, the old outline is stale and the decoration may land on a line that no longer holds the same step, but the run completes rather than failing.

Fail-soft error handling

• When the CLI path does not resolve to an executable, the spawn fails. The run handler catches the error, marks the file and all its steps errored with a diagnostic message, and shows a one-time warning notice (per extension activation) mentioning the vouchfx.cliPath setting.
• When the CLI exits with a non-zero status and produces no events, the items are marked errored with a message including the exit code and stderr output (truncated to 800 characters for inline display).
• When the events file is unreadable or contains no usable events (e.g., the CLI exited 0 but wrote no JSON Lines), items are marked errored with a message explaining that no events were produced.
• The run handler is wrapped in a try-catch that swallows all unexpected errors; Test Explorer integration never throws and never regresses the schema authoring experience.

11. Embedded C# Intelligence

Schema validation makes the declarative YAML safe, but it stops at the boundary of a script step. The moment an author writes C# inside a script step, schema validation has nothing to say. The extension provides syntax colouring for embedded C# and records full IntelliSense as a documented fast-follow.

11.1 C# syntax highlighting (delivered)

Delivered: A TextMate injection grammar (syntaxes/csharp-in-e2eyaml.injection.json) embeds VSCode's built-in C# grammar into the block scalar that follows a code: key in a script.csharp step. This provides syntax highlighting (colours for keywords, strings, member access, etc.) while the surrounding YAML remains YAML-coloured. The highlighting runs entirely inside the TextMate tokeniser and requires no language service, project, or network access. It does not provide completion, type checking, or diagnostics — those features are recorded as a documented fast-follow (see section 11.3).

11.2 C# IntelliSense (documented fast-follow)

Deferred (documented fast-follow): Full embedded C# IntelliSense — completion, type checking, diagnostics, and go-to-definition on members of Vars, Services, Secrets, and Webhooks — is a documented fast-follow, not delivered in v1. The intended approach and concrete technical reasons for the deferral are recorded in tools/vscode-vouchfx/docs/csharp-intellisense.md. The barrier is not scope but technology: the modern C# Dev Kit / Roslyn LSP does not cleanly support virtual-document completion forwarding for in-memory .cs files not bound to an MSBuild project, and YAML ↔ virtual-document position mapping is non-trivial; the robust long-term solution is a custom Roslyn language-server host, a substantial multi-release effort. Until then, syntax highlighting plus YAML schema validation is the supported v1 authoring experience.

11.3 Planned approach for C# IntelliSense

When the fast-follow is scheduled, the preferred path is an embedded "virtual document": the extension synthesises an in-memory .cs by concatenating a preamble that declares the engine's global variables (Vouchfx.Engine.Abstractions.ScriptGlobalVariables — Vars, Services, Secrets, Webhooks) with the author's block-scalar body, then forwards completion and diagnostic requests to the installed C# language server and maps positions back into the .e2e.yaml document. The planned approach and technical trade-offs are fully documented in tools/vscode-vouchfx/docs/csharp-intellisense.md.

11.4 Language support and accessibility commitments

The extension's user-facing strings — diagnostic messages, command labels, hover documentation — are English-only at v1.0. The codebase is structured so that these strings are localisable rather than hard-coded literals, so a future localisation effort is configuration rather than rewrite, but no second language is translated for v1.0. The Accessibility commitments stated in the architecture blueprint's reporting section (§14.11) apply equally to the extension's surfaces: WCAG 2.1 AA for the inline-decoration colour scheme (paired with shape or text for colour-blind users), keyboard navigation through the Test Explorer panel, and semantic markup in any HTML the extension renders.

12. Semantic Project Validation

The most valuable feature of the extension is also the most ambitious. Schema validation confirms that a test file is well-formed; semantic validation confirms that it is true to the project around it. It catches the class of error where the YAML is perfectly valid but refers to something that does not, or no longer, exists.

12.1 What semantic validation checks

Through the Language Server Protocol, the extension parses the workspace and cross-references the test file against the project's contracts. It compares an http step's path and method against the service's OpenAPI specification, and flags a call to an endpoint that the specification does not define. It checks an mq-publish or mq-expect step against the project's AsyncAPI contracts and the registered Avro schemas, and flags a payload that violates the schema for that topic. It checks a db-assert step against the known SQL schema, and flags a query against a table or column that does not exist.

Test construct	Validated against	Error caught
http step path and method	The service's OpenAPI / Swagger specification.	A call to a REST endpoint that the API does not expose.
mq-publish / mq-expect payload	AsyncAPI contracts and registered Avro schemas.	A message that violates the schema for the topic.
db-assert query	The database's SQL schema.	A reference to a table or column that does not exist.
environment logical names	The names declared in the file's own environment section.	A step targeting a dependency that was never declared.

Table 12.1 — Semantic checks performed by the language server.

12.2 Shift-left validation

The effect of these checks is to move error detection as far left as it can go — from a failed test run, back to the moment the author types. An endpoint that was renamed in the API is flagged in the editor the next time the test file is opened, rather than surfacing as a confusing failure in CI an hour later. This is the same shift-left philosophy the agentic Healer applies at runtime, but applied at authoring time: the cheapest place to catch a broken reference is before the test ever runs.

12.3 Keeping contracts current

Semantic validation is only as accurate as the contracts it reads. The extension watches the workspace for changes to OpenAPI, AsyncAPI, and schema files and re-validates open test files when they change. For contracts that live outside the workspace — a schema registry, a shared API catalogue — the extension can be configured with the relevant endpoints so that validation reflects the authoritative source rather than a stale local copy.

13. Run, Debug, and Authentication

Authoring is only half of the loop; the author also needs to run the test and read its result without leaving the editor. This section specifies the execution-facing features of the extension and how they behave across the three deployment topologies.

13.1 Running a suite from the editor

The extension contributes commands and code lenses that let an author run a single test, or a whole file, directly from the editor. Execution invokes the same test runner that a CI pipeline would use, so a test that passes from the editor passes in the pipeline. Results are surfaced inline: each step is annotated with its outcome, and a failed assertion is shown against the step that produced it, using the verdict taxonomy from the architecture document so that a genuine failure is visually distinct from an environment error.

13.2 Debugging script steps

Because script steps contain real C#, they can be debugged. The extension integrates with the .NET debugger so that an author can set a breakpoint inside a script step, inspect the shared variable context at that point, and step through the embedded code. Debugging a declarative step is not meaningful — there is no imperative code to step through — but the extension makes the full state of the variable context inspectable after every step, which serves the same diagnostic purpose.

13.3 Authentication in the enterprise topology

In the enterprise topology, running a suite against the cloud backend requires an identity. The extension therefore participates in the SSO flow described in the architecture document: it authenticates the author against the enterprise identity provider via SAML or OIDC and obtains a short-lived JSON Web Token. That token authorises the provisioning of container topologies and carries the author's RBAC claims. The practical consequence inside the editor is that a developer without permission to launch a large load test is told so at authoring time, rather than discovering it through a failed run.

13.4 Choosing the execution target

The extension lets the author choose where a run executes — against the local Docker socket, against the SaaS cloud, or against an enterprise backend — and remembers the choice per workspace. Because the architecture guarantees topological parity, the test file does not change between targets; only the runner's configuration does. This makes it natural for a developer to iterate locally for speed and then run the identical suite against the cloud before merging.

13.5 Running tests from the command line

The vouchfx command-line tool discovers and runs .e2e.yaml test scenarios from the shell, optionally filtering them by metadata and file changes. It discovers all scenarios under a given path (recursively), applies the optional selection filters, compiles and executes them against an orchestrated topology, and returns an exit code reflecting the overall verdict.

vouchfx run [<path>] [--tag <tag>...] [--owner <owner>...] [--path <glob>] [--changed-since <ref>] [--parallel <n>] [--watch]

The positional <path> argument specifies the directory to search for scenarios (defaults to . if omitted). All selection and execution options are optional and composable — a scenario is selected if it matches all supplied criteria (AND across dimensions):

Option	Meaning
--tag <tag>	Repeatable. Selects scenarios whose metadata.tags contains any of the supplied tags (OR within the dimension).
--owner <owner>	Repeatable. Selects scenarios whose metadata.owner equals any of the supplied owners (OR within the dimension).
--path <glob>	Selects scenarios whose absolute file path matches the supplied glob pattern. Supports * (matches within a path segment), ** (matches across segments), and ? (single character). A pattern with no wildcard characters is matched as a substring.
--changed-since <ref>	Selects only scenarios whose file has changed since the given git reference (as determined by git diff <ref>...HEAD plus the dirty working tree). Requires a git repository.
--parallel <n>	Run up to N scenarios concurrently, each owning its own container topology. Opt-in parallelism: each concurrent scenario multiplies container cost, so omitting this flag runs scenarios sequentially against one shared topology (the default). Must be 1 or greater. Cannot be combined with --watch.
--watch	Run once, then watch the .e2e.yaml file and re-run automatically on save. Re-uses the already-built container topology while the environment block is unchanged; rebuilds the topology only when the environment changes. Useful for fast local iteration on a single file. Press Ctrl-C to stop. Cannot be combined with --parallel.

Delivered: Exit codes follow the verdict taxonomy, with opt-in flags to gate CI on infrastructure errors and timeouts:

Exit code	Meaning	How to trigger
0	Success — all scenarios passed, or only environment errors / inconclusive (the default, only Fail breaks CI by default).	Default. All scenarios Pass, or only EnvironmentError / Inconclusive and no opt-in flags set.
1	Test failure — at least one scenario failed (the test assertions did not hold).	Default; always exits 1 on Fail.
2	Usage error (bad arguments, missing directory, invalid --changed-since ref).	Default; returned by System.CommandLine on parse errors.
3	Environment error — the aggregate verdict was EnvironmentError and the author opted in.	Set --fail-on-env-error to exit 3 instead of 0 when infrastructure breaks (container fails to start, tunnel collapses, etc.).
4	Inconclusive — the aggregate verdict was Inconclusive and the author opted in.	Set --fail-on-inconclusive to exit 4 instead of 0 on timeout or unmet capture dependency.

Examples:

# Run all scenarios under the current directory
vouchfx run

# Run scenarios in a specific directory
vouchfx run ./tests/e2e

# Select scenarios by tag (any of the tags match)
vouchfx run --tag smoke --tag integration

# Select by owner and path together (all match)
vouchfx run --owner alice --path "**/orders/*"

# Select only scenarios that changed since the last commit
vouchfx run --changed-since HEAD~1

# Combine multiple filters
vouchfx run ./tests --tag integration --owner team-a --changed-since main

# Watch a single file for changes and re-run automatically
vouchfx run ./tests/users.e2e.yaml --watch

# Run scenarios in parallel (two at a time, each with its own topology)
vouchfx run --parallel 2

# Produce an HTML report (self-contained, standalone, with reproducibility envelope)
vouchfx run --html report.html

# Produce JUnit XML for CI aggregation
vouchfx run --junit results.xml

# Combine: run, fail on environment errors, write both reports
vouchfx run ./tests --fail-on-env-error --html report.html --junit results.xml

14. Result Reporting from the Author's Perspective

Authoring a test and running a test are not the end of the loop; reading what the test produced is. This section specifies what an author of YAML tests sees when a suite completes — the renderings, the verdicts, the diagnostic stories — because the value of every previous chapter of this document is realised only at the moment a developer understands why a step turned red. The engineering view of the same reporting layer, including the structured event stream that drives every renderer, is in Section 14 of the companion Technical Architecture & Engineering Blueprint; this section deliberately reads the report back through the eyes of the YAML author.

14.1 The verdicts an author will see

Every step and every scenario in the DSL terminates with exactly one of four verdicts. They are not interchangeable, and reading them as if they were is the single fastest way to misunderstand a red build. The four are: pass — the step's assertions held; fail — the step's assertions did not hold, so the system under test produced an outcome different from the one declared in the YAML; environment error — the step could not be evaluated because surrounding infrastructure failed, for example a container that did not start or a tunnel that collapsed; and inconclusive — evaluation began but was interrupted before a verdict could be reached, typically because a dependency on an earlier step's capture was unmet. The author sees these in distinct colours and distinct sections of the report, with environment errors and inconclusive verdicts counted separately from failures.

14.2 The polling timeline for asynchronous steps

This is the rendering authors will appreciate most, because it is what most existing testing tools refuse to give them. When a step running in verifyMode: RETRY fails, the report does not say “the step timed out.” It shows every attempt the engine made: when the attempt happened relative to the step's start, what the engine observed, and how the observation differed from what the YAML expected. The author reads the asynchronous story the way they wrote it, attempt by attempt, instead of inferring it from raw broker logs.

Step  expect-billing-event  (mq-expect.kafka)        FAIL  (timeout after 30.0s)

      t=  0.5s   attempt 1   no message matched the key u-8af2-c13e
      t=  1.5s   attempt 2   1 message matched, payload.status was "PROCESSING"
                             expected "PENDING"
      t=  4.0s   attempt 3   no message matched
      t= 10.0s   attempt 4   no message matched
      t= 20.0s   attempt 5   no message matched
      t= 30.0s   attempt 6   timeout fired

Reading this rendering, the author can tell at a glance that the system produced a matching event briefly and then stopped — the signature of a retry-and-overwrite upstream rather than a missing event. That single distinction would otherwise require an hour of log archaeology, and is the difference between a useful integration testing tool and an annoying one.

14.3 The captured-variable thread through a scenario

A scenario is a sequence of steps whose later steps depend on values captured by earlier ones. When a step fails, the report renders the full thread of state through the scenario — what each step captured, what each step substituted, and where any value originated — in the same vocabulary the author wrote it in. The thread is what lets the author trace a failure backward from where it surfaced to where it actually began, often two or three steps earlier in the file.

Scenario  users-e2e

  1. create-user             PASS (203ms)
       captured  newUserId       = "u-8af2-c13e"   (from $.id in the response)

  2. expect-billing-event    FAIL (30024ms)
       substituted  newUserId    = "u-8af2-c13e"   (from step 1)
       captured     billingAccountId = (none — step failed before capture)

  3. assert-projection       INCONCLUSIVE (skipped)
       reason  depends on billingAccountId from step 2, which was not captured

Notice the third step's verdict. It is not a failure; it is inconclusive. Step 3 depends on a value step 2 never produced, so reporting it as a failure would falsely accuse the database of misbehaviour. Rendering this distinction prevents a single root-cause failure from cascading into a wall of red unrelated to the real defect, which is one of the most common failure modes of integration testing reports as a category.

14.4 The expected-versus-observed diff

For every assertion in a failing step, the report shows what the YAML expected and what the engine observed, side by side. The diff is rendered in the vocabulary of the data the step worked with: a db-assert.postgres failure shows row counts and column-by-column field comparisons; a db-assert.mongodb failure shows a document diff with nested paths; a db-assert.redis failure shows key-and-value comparisons; an http.rest failure shows expected status, body fields, and headers against what came back. Each provider supplies its own rendering hook, which is why the diff stays faithful to the technology rather than collapsing into a generic textual comparison.

14.5 The reproducibility envelope on every report

Every report carries the context needed to reproduce the run elsewhere: the engine version, the schema version, the list of providers and their versions, the resolved container image digests, the seeded variable values, and the content hash of every YAML file. The author can share a failed report with a teammate and that teammate can reproduce the exact failure on a different machine without guessing. This eliminates the “works on my machine” debate before it begins; it is also the artifact an auditor requires in regulated industries.

14.6 Where the author sees the report

The same report appears in three places, fed from the same underlying event stream so that one place cannot drift from another.

Surface	What the author sees there
Terminal (the CLI runner)	A tree-structured renderer with colour, the polling timeline, and the captured-variable thread; designed for the developer iterating locally and for the on-call engineer triaging from another machine. Exit codes are deterministic so CI gates work.
VSCode (the extension)	Inline decorations on the failing YAML line, an integrated panel showing the per-step records, and the live execution feed during a long suite. Clicking a failure jumps the cursor to the step that produced it.
HTML file written to disk	A standalone report attachable to a CI build or shared as a link, with the full reproducibility envelope embedded. Stable across browsers and printable for record-keeping.
JUnit XML	Machine-readable output for CI systems that aggregate test results across many tools; not for human consumption.

The Team and Enterprise tiers add a hosted cross-run dashboard with flakiness scores and trend charts, RBAC on report access, audit-grade immutability, and SIEM export. From the YAML author's perspective these are extensions of the same report rather than separate surfaces — a failure in the local terminal is the same record that appears, later and aggregated, in the dashboard a team lead reads on Monday morning.

14.7 What the report does not contain

The report contains links to logs, not logs themselves. Engine telemetry — traces, container logs, broker state, performance load profiles — lives in the observability stack, and each correlation id in the report resolves to a span or a log bundle there. Keeping the report a navigable index rather than a duplicate of the telemetry stack is what keeps it fast to render and small enough to attach to a build. An author who needs to dig deeper follows the link; the report does not pre-fetch what most readers will never need.

If you read only one rendering, read the polling timeline

The polling timeline of section 14.2 is the rendering that most cleanly repays the platform's investment in asynchronous-first design. Existing integration testing tools rarely have a native concept of “the message arrived 4 seconds late” or “the projection appeared briefly and then vanished,” so they report a generic timeout. The platform records every RETRY attempt as a structured event and renders the resulting timeline directly, turning asynchrony from a forensic problem back into a story the test author can read.

15. Design Risks and Open Questions

Both the language and the extension carry design risks worth stating plainly, so that they can be tracked rather than discovered late. Each risk below is paired with the current mitigation and the question that remains open.

Risk or open question	Current position
Mapping diagnostics from extracted C# fragments back to the right YAML line is error-prone.	Embedded-language support with explicit position mapping; this is the extension's hardest engineering task and needs a dedicated test corpus.
Authors may overuse script steps and erode the DSL's readability and validatability.	Documented guidance and a future linter rule that flags files dominated by script steps; not yet enforced.
Semantic validation depends on contracts that may be missing, stale, or external.	The extension watches local contracts and can be pointed at external sources; behaviour when a contract is simply absent must degrade to a warning, not a hard error.
Schema evolution could break existing test files.	Versioned schema with per-file targeting; a migration tool for moving files between versions is an open item.
Offline and air-gapped enterprise networks cannot fetch a hosted schema.	Local and internal schema resolution is supported; the default configuration for air-gapped installs needs to be specified.
The language currently models a single linear sequence of steps.	Sufficient for the MVP; conditional and parallel step groups are deferred and should be designed before they are added piecemeal.
Inconsistent provider naming or field conventions across the catalogue confuses authors.	Reserved registry of family names governed by the platform team; a style guide in CONTRIBUTING.md that defines casing, common field names, and the use of target and expect; review by core maintainers at Vouched badge award.
Schema-fragment quality varies between Community providers, weakening the editor experience.	A schema-fragment validation fixture in the official integration matrix that every Vouched provider must pass; the Vouched status is surfaced in the editor and in diagnostics so authors know what they have installed.
A provider supplies a poor failure-diff renderer, producing reports authors cannot read.	An engine-provided default diff renderer covers any provider that omits its own; the Vouched badge review includes inspecting the diff against a corpus of representative failures.
Authors mistake an inconclusive verdict for a failure (or vice versa) and chase the wrong defect.	Distinct colour, distinct counter, and distinct documentation; the inconclusive verdict's reason line is mandatory in the report and tells the author which earlier step's capture was unmet.
Authors inline a literal credential into a field rather than using a secret reference.	The ${secret:…} syntax is documented as the only sanctioned way to supply credentials; a future lint rule flags secret-shaped literals in test files; report redaction limits the blast radius if one slips through.
A seed fixture drifts out of step with the schema of the dependency it loads.	Seeds are applied inside the health-gated lifecycle and a failed seed is an environment error with a clear message; fixtures live in source control beside the test so they are reviewed and versioned together with it.

16. Appendix: A Complete Reference Test File

This appendix assembles the constructs defined throughout the specification into a single, complete test file. It is the platform's reference scenario: registering a user, confirming the resulting Kafka event, confirming the database projection, and confirming the outbound webhook — a transaction that crosses four technologies in one declarative file. It now also shows the seed block establishing reference data and a secret reference supplying a credential without exposing it.

metadata:
  name: "User registration end-to-end"
  owner: "payments-team"
  tags: [smoke, billing]

environment:
  services:
    orders-api:
      type: dotnet-service
      project: Orders.Api
      env: { ApiKey: "${secret:vault/orders-api/key}" }
  dependencies:
    orders-db:  { type: postgres, version: "16" }
    events:     { type: kafka, schemaRegistry: true }
  seed:
    orders-db:
      sql: [ "fixtures/reference-plans.sql" ]

variables:
  tenantId: "acme-corp"
  basePath: "/api/v1"

steps:
  - id: create-user
    type: http.rest
    target: orders-api
    method: POST
    path: "{basePath}/users"
    body: { email: "jane@{tenantId}.example", plan: "standard" }
    expect: { status: 201 }
    capture: { newUserId: "$.id" }

  - id: expect-billing-event
    type: mq-expect.kafka
    target: events
    topic: billing.account.created
    verifyMode: RETRY
    timeout: 30s
    match: { key: "{newUserId}" }
    capture: { billingAccountId: "$.payload.accountId" }

  - id: assert-projection
    type: db-assert.postgres
    target: orders-db
    verifyMode: RETRY
    timeout: 20s
    query: |
      SELECT plan, billing_account_id
      FROM users WHERE id = @userId
    parameters: { userId: "{newUserId}" }
    expect:
      rowCount: 1
      row: { billing_account_id: "{billingAccountId}" }

  - id: expect-webhook
    type: webhook-listen.http
    pathExpose: "/hooks/welcome"
    verifyMode: RETRY
    timeout: 30s
    expect:
      method: POST
      body: { userId: "{newUserId}" }

— End of the YAML DSL Specification & VSCode Extension Design. The companion document is the Technical Architecture & Engineering Blueprint.