vouchfx Troubleshooting

This guide covers real failure modes, what they mean, and how to fix them.

Quick index: - Docker is not running or not reachable - Transient image pull corruption: "short read" or "unexpected EOF" - EnvironmentError: HealthGate timeout of 00:00:20 - Every port-publishing container fails at startup ("should have valid address at this point") - Private registry and air-gapped operation - Discovery root does not exist (dotnet run path resolution gotcha) - Script body, document, and YAML expansion limits - Understanding the four verdicts - Secret leakage in exception messages - Build fails with warnings-as-errors - Aspire topology timeouts or hangs - Steps run but assertions fail - Capture fails or placeholder is empty - Kafka messages not consumed (ordering or timing) - Validation errors at authoring time - Git selection diagnostics

---

Docker is not running or not reachable

Symptom:

RunSuiteAsync: topology failed to start - Cannot connect to the Docker daemon at unix:///var/run/docker.sock

or

Cannot connect to the Docker daemon at unix:///var/run/docker.sock

What it means: vouchfx orchestrates containers via Docker (Aspire + Testcontainers). If the Docker daemon is not running or not reachable, the topology cannot start.

Fix:

1. Check Docker is running:

   docker info
   

   You should see version and runtime information. If it fails, start Docker.

2. On Windows / macOS: Start Docker Desktop.

3. On Linux: Ensure the dockerd daemon is running:

   sudo systemctl start docker
   

4. If Docker is running but still unreachable: Check permissions. On Linux, the current user must be in the docker group:

   sudo usermod -aG docker $USER
   # Then log out and back in for the group change to take effect
   newgrp docker
   

5. Verify the socket exists:

   # Linux/macOS
   ls -la /var/run/docker.sock
   
   # Windows (Docker Desktop with WSL2)
   # The socket is exposed via npipe; Docker Desktop exposes it at the standard locations
   

In CI: Ensure the CI runner has Docker available. GitHub Actions' ubuntu-latest includes Docker by default. For GitLab, use docker:dind service or a socket-bind mount (see README.md § CI integration with GitLab CI).

---

Transient image pull corruption: "short read" or "unexpected EOF"

Symptom:

ERROR: failed to build: failed to solve: failed to compute cache key:
short read: expected 12108405 bytes but got 11522155: unexpected EOF

or a docker pull / topology start failing with unexpected EOF or failed to register layer.

What it means: A layer download was corrupted mid-transfer — a registry or network hiccup, not a vouchfx or Dockerfile defect. When it happens during topology start (DCP's cold pull of a dependency image), vouchfx reports it as an EnvironmentError — per the verdict taxonomy, infrastructure breakage, never a test failure. When it happens while building your own application image, the build fails before vouchfx runs at all.

Fix:

1. Retry the pull or build — a plain retry almost always succeeds; the corrupted partial layer is discarded and re-fetched.
2. If it persists, pull the base image explicitly to isolate the failing layer:

   docker pull <image>
   

   then check free disk space, and clear a corrupt build-cache entry with:

   docker builder prune
   

3. In CI, re-run the job. To reduce exposure on cold runners, pre-warm the images the suite needs (the reusable workflow's prewarm-images input does this; each pull is best-effort and non-fatal).

---

The Aspire orchestration component is not installed / could not be located

Symptom:

System.IO.FileNotFoundException: The Aspire orchestration component is not installed
at "/home/runner/.nuget/packages/aspire.hosting.orchestration.linux-x64/13.4.2/tools/dcp".

or, from legacy cache-based builds and unbundled source/custom AppHosts:

EnvironmentError: The Aspire DCP orchestration component could not be located. …
no 'aspire.hosting.orchestration.<rid>' version '13.4.2' package was found at '…'.

What it means: the inline Aspire environment could not locate a matching native DCP tool. Published 1.0.0 distributions bundle Aspire 13.4.2 under dcp/13.4.2/<rid>/tools/, beside the CLI. Supported process platforms are win-x64, glibc linux-x64, osx-x64 and osx-arm64.

Fix for published 1.0.0 distributions: reinstall the complete matching package/archive. Keep the dcp/ tree with the CLI; copying only the executable is insufficient. An incomplete declared bundle fails explicitly, even when another DCP happens to exist in a local NuGet cache. On Unix, archives and installers must preserve executable permissions on dcp and dcptun_c; global-tool startup repairs those bits in its selected bundle. A read-only installation with missing permissions needs repair by its owner. Unsupported RIDs require a supported distribution or an explicit matching DCP override.

Legacy published rc.6 and older cache-based builds: the native tools are not bundled. A fresh-cache 1.0.0-rc.6 installation was measured on 2026-10-10 to fail before Docker starts; an earlier cache fallback did not eliminate this prerequisite. Supply the exact platform's Aspire 13.4.2 tools and set ASPIRE_DCP_PATH to their directory, or upgrade to the latest stable CLI. Do not infer the installed version from its error path: even newer legacy packages can display a packaging runner's stale path. Restoring an AppHost at the exact version populates tools for source builds, but is not a prerequisite for installed 1.0.0 distributions. The retired Aspire workload does not provide these NuGet tools.

$env:ASPIRE_DCP_PATH = "$env:USERPROFILE/.nuget/packages/aspire.hosting.orchestration.win-x64/13.4.2/tools"

The explicit override remains authoritative; unset it to use the installed bundle. DCP provisioning failures remain EnvironmentError. CI should pass --fail-on-env-error --fail-on-inconclusive and check that scenarios actually executed. See the portability incident.

---

EnvironmentError: HealthGate timeout of 00:00:20

Symptom:

RunSuiteAsync: topology failed to start - HealthGate timeout of 00:00:20 on resource 'postgres'

What it means: This timeout is not the vouchfx 120-second outer gate — it is an Aspire/DCP internal per-resource watchdog (approximately 20 seconds per resource). When a resource takes longer than 20 seconds to become healthy (e.g., a large image pull on a cold cache, or a slow container startup), DCP's watchdog expires.

This is NOT a vouchfx configuration knob (there is no --health-check-timeout flag). The 20-second window is built into Aspire/DCP and cannot be extended at runtime.

Fix:

1. Pre-warm Docker images locally before running the suite (most common fix):

   # On your machine or in CI, before vouchfx runs:
   docker pull postgres:16
   docker pull myco/order-service:latest
   

   Once an image is in the local Docker cache, DCP skips the pull and starts the container much faster.

2. In CI, use the prewarm-images workflow input (GitHub Actions):

   vouchfx-e2e:
     uses: tomas-rampas/vouchfx/.github/workflows/vouchfx-run.yml@<commit-sha>
     with:
       scenario-path: ./tests/e2e
       prewarm-images: |
         postgres:16
         myco/order-service:latest
   

Or for GitLab CI, set VOUCHFX_PREWARM_IMAGES:

vouchfx-run:
  variables:
    VOUCHFX_PREWARM_IMAGES: |
      postgres:16
      myco/order-service:latest

1. Check container startup performance locally. If a container is consistently slow to start, it may have an expensive initialization step (e.g., schema migration, data load). Consider:
2. Running the initialization ahead of time (in the container image build, not at startup).
3. Using a healthcheck endpoint that returns quickly (not one that validates database connectivity, which adds latency).

4. Profiling the container with docker logs to see where time is spent.

5. Serialize topology startup if you are running tests in parallel locally. The [assembly: CollectionBehavior(DisableTestParallelization=true)] attribute (on xUnit test projects that use Aspire) serialises test startup, preventing concurrent DCP resource startup from overwhelming the machine and triggering timeouts. This is already applied in the vouchfx test projects (see Vouchfx.Engine.Runtime.Tests for the example).

6. In CI, increase runner capacity or use a faster image pull network. If your CI runner has constrained network or disk bandwidth, image pulls take longer. Use a faster runner if available, or pre-cache the image in the runner's local Docker registry.

Why this happens: Aspire's Distributed Cloud Provisioning (DCP) watches each resource and gives it ~20 seconds to become healthy. This is a safety net to prevent tests from hanging indefinitely. If multiple resources start concurrently, they compete for I/O (disk, network), and a large image pull can exceed 20 seconds. Pre-warming the cache is the most reliable fix.

---

Every port-publishing container fails at startup ("should have valid address at this point")

Symptom: the topology never comes up, and the logs carry one or both of these lines:

warn: Unable to allocate a network port for service 'my-broker-https'; service may be unreachable
      and its clients may not work properly.
fail: System.IO.InvalidDataException: Service my-broker-https should have valid address at this point
      at Aspire.Hosting.Dcp.DcpModelUtilities.TryAddLocalhostAllocatedEndpoint(...)

Every service or dependency that publishes a container port fails; a suite that publishes none still runs, and a plain docker run -p 0:80 nginx:alpine still publishes a port perfectly well.

What it means — and the message points at the wrong layer. Nothing is wrong with ports. DCP's controller host fails to start: it refuses its state-store directory because that directory's owner does not match the current user or token owner, and exits with code 1 about 130 ms in. With the controller dead nothing allocates ports, so Aspire waits for a port allocation that will never arrive and gives up on a fixed 60-second timeout — twice, which is the constant ~2 minutes you wait before the throw. Unable to allocate a network port is Aspire's downstream wording for "the allocation never came", not a description of the fault. It is not a defect in your suite, not port exhaustion, and not transient: re-running will fail identically. Full evidence in issue #420.

The fix (Windows):

1. Look in your ~/.dcp folder. It holds one state-store directory per privilege level: state for ordinary runs and state.elevated for elevated ones.
2. Check the owner of each (Get-Acl ~\.dcp\state.elevated | Select-Object Owner). The fault is one of them being owned by somebody other than you — BUILTIN\Administrators rather than your account, in the case that was diagnosed.
3. Rename or delete the offending directory, then run from a non-elevated shell. Verified: renaming it aside took the previously-failing suite from a 2-minute failure to green in 23 seconds on the same host.

Deleting it is not enough on its own if you keep running elevated. Measured on the same host afterwards: an elevated run recreated state.elevated and Windows gave the new directory to BUILTIN\Administrators rather than to the running account, so DCP's own ownership check refused it again and the fault returned immediately. If you must run elevated, take ownership of state.elevated after DCP creates it rather than expecting a delete to stick:

takeown /f "$env:USERPROFILE\.dcp\state.elevated" /r /d Y

Use $env:USERPROFILE, not %USERPROFILE%: PowerShell does not expand cmd-style %VAR% in the arguments it passes to a native executable, so takeown would receive the eleven literal characters and fail. /r recurses; /d Y pre-answers the per-directory prompt that recursion raises. From cmd.exe the %USERPROFILE% form is correct and the quotes are unnecessary.

Epistemic note, since this page labels the rest: the diagnosis is measured — the ownership refusal, DCP's exit code, the 60-second waits, and the fact that an elevated re-run recreates the directory under BUILTIN\Administrators. The takeown remedy is inferred from that mechanism, not executed: what was verified end to end is renaming the directory aside and running non-elevated, which took the failing suite to green in 23 seconds. If you run the takeown form, please say on issue #420 whether it held.

Why it looked intermittent. The two state stores are per-privilege-level, so an elevated shell and an ordinary one use different directories — only one of which may be broken. A fault that follows how you happened to launch the terminal looks like it comes and goes on its own. The mechanism is measured (the directory selection and the ownership of the recreated directory were both observed directly); attributing the earlier sessions' apparent intermittency to it is inferred, because nobody recorded whether those shells were elevated.

winnat and Docker Desktop restarts were the previously-suggested remedies. They are not required and do not address this cause; skip them.

The engine captures the diagnostics for you — and that is how this was diagnosed. Raising DCP's log level after the fault appears had failed twice, because the fault stopped reproducing before the instrumented run started. So the capture is always armed instead. Whenever a topology fails to become ready, vouchfx writes the DCP log traffic it buffered during that start — including the Debug-level line quoted above, which never reaches the console — to a per-user file:

Platform	Capture file
Windows	%LOCALAPPDATA%\vouchfx\dcp-capture-<utc-timestamp>.log
Linux, macOS	~/.local/share/vouchfx/dcp-capture-<utc-timestamp>.log

On Linux the root follows XDG_DATA_HOME when you have set it, exactly as every other application using that convention does; ~/.local/share is the fallback, and is where the file lands on an unconfigured desktop or CI runner. Whether macOS honours XDG_DATA_HOME here is unverified — the engine asks .NET for the per-user local application data directory and does not implement the rule itself, and no macOS host was available to measure. ~/.local/share is the expected location there either way; if you have XDG_DATA_HOME set on macOS and find the capture somewhere else, that is worth reporting.

The twelve most recent captures are kept; older ones are pruned automatically. Twelve rather than a handful because #420's last occurrence was eight consecutive failures in one session, and retention below that would delete the very captures the session produced.

The 1.0.0 CLI also writes a separate private application-model context header. The engine snapshots registered resource names before DCP starts, retaining at most 64 resources and 128 characters per name, and records omitted-resource and truncated-name counts. This header has its own 16 KiB bound. It helps identify the host's resource even when upstream initial-creation logs omit names. It reads no environment or annotation values, remains separate from actual native DCP entries, and never enters console warning tails.

The capture is written whenever a topology fails to become ready — the start itself, the health gates that follow it, service discovery, the secured-endpoint probe and the seed — because this fault can surface at any of them. On a platform with no per-user directory nothing is written and the Environment error says so rather than falling back to a shared temporary directory.

The reported Environment error names the capture file on most of those paths, but not on all of them, and it is worth knowing which. When the failure is one the engine classifies itself — the topology start, a health gate, or service discovery — the error names the file as a platform token and quotes the last few warning lines inline, so the evidence survives in a CI log even when the runner's filesystem does not. A failure in the secured-endpoint probe or the seed reports its own message instead: those two build their error where they raise it, before the capture is written, so the file exists but nothing in the error points at it. If a run fails at either and you want the DCP traffic, look in the directory above — the newest capture is that run's.

On CI, retain and upload captures before the runner disappears. A build agent's filesystem is discarded when the job ends. For your own suites, set VOUCHFX_DCP_CAPTURE_DIR to a directory inside the workspace, and add an upload step for it — the reusable vouchfx-run.yml workflow uploads results.xml and report.html only, so captures are not carried out unless you upload them yourself:

env:
  VOUCHFX_DCP_CAPTURE_DIR: ${{ github.workspace }}/vouchfx-captures

# ... and, after the run:
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
  if: always()
  with:
    name: vouchfx-dcp-captures
    path: vouchfx-captures/
    if-no-files-found: ignore
    retention-days: 7

This repository's Docker/Aspire CI keeps the default per-user root so its READY-path guard remains meaningful. It snapshots before the test, then collects new matching captures even after a failed test and uploads them as integration-dcp-captures with seven-day retention. Existing files are excluded; a zero-capture run uploads only its count manifest. Collection refuses unsafe paths, links and excessive payload, and upload requires successful collection. This does not recover captures the engine has already pruned.

A CI artefact is as public as an issue comment. Actions does not mask secrets inside uploaded artefact contents, and on a public repository anyone can download them. Read the section below on what a capture can contain before uploading one from a job whose suite routes any credential in via ${env:NAME}.

The path is used exactly as given (no vouchfx subdirectory is appended) and must be absolute — a relative value is refused rather than resolved against the working directory, and the Environment error says so rather than quietly writing to the per-user directory instead. Note that a directory you choose keeps the permissions it already has — vouchfx narrows permissions only on a directory it creates itself, and never on one you already had. The owner-only default applies to the per-user location.

When the capture shows this specific refusal, the reported Environment error also carries the one-line cause and remedy inline, so you do not have to open the file to know what to do. If it does not — a different DCP failure, or a reworded one — the file is where the answer will be.

If you meet a variant of this, please attach the capture to issue #420 — with a skim first. A capture is a verbatim record of what Aspire logged while the topology was coming up, which includes container specifications and therefore container environment variables. Concretely, it can hold Aspire's generated per-run passwords for managed dependencies (throwaway, valid only for that run's container and destroyed with it), the fixed local test credentials your suite declares, any host environment value you routed in with ${env:NAME}, and absolute paths on your machine. It cannot hold a ${secret:...} value: the engine refuses that reference outright in both services[].env and dependencies[].env, so no resolved secret ever reaches a container specification for this layer to log.

Everything in that list is already visible to docker inspect for the same containers on the same machine, so the file adds no local exposure you did not already have — but attaching it to a public issue publishes it, which is why it is worth a skim rather than a warning to keep it to yourself.

The warning lines quoted inline in the Environment error are a separate question. They are part of that error's detail and pass through the engine's ordinary secret-redaction path, but they arrive there truncated and with any non-ASCII character replaced, and that redaction matches values exactly — so treat the inline tail as best-effort redacted rather than guaranteed. The capture file is not redacted at all.

A successful start writes nothing at all: the buffer is discarded the moment the topology is up, so a healthy run leaves no file and produces no extra output. If you need to switch the recorder off entirely, set VOUCHFX_DCP_CAPTURE=0 (that exact value; anything else leaves it armed).

---

Private registry and air-gapped operation

Symptom: Your organisation uses a private or internal container registry, and you need to run vouchfx tests without pulling images from Docker Hub or public registries. Alternatively, you work in an air-gapped environment where containers are pre-warmed locally and must not attempt external pulls.

What it means: By default, vouchfx pulls container images from public registries (Docker Hub for unqualified names, or directly from the specified registry host). Teams on a private registry (Nexus, Artifactory, ECR, ACR) or in regulated/air-gapped environments need to redirect those pulls or enforce local-cache-only operation. The platform provides two complementary mechanisms: environment-level registry redirection (imageRegistry) for un-qualified images, and per-dependency image override (image: field) for explicit control.

Fix:

1. Redirect public images to an internal mirror (imageRegistry)

If your organisation mirrors public images on an internal registry, use the imageRegistry environment-level override to redirect un-qualified references:

environment:
  imageRegistry: nexus.corp.local/docker-mirror

  services:
    my-api:
      image: myco/myapi:latest           # Will be redirected to nexus.corp.local/docker-mirror/myco/myapi:latest

  dependencies:
    db:
      type: postgres
      # Will use nexus.corp.local/docker-mirror/library/postgres:18.3 (Docker's library namespace; Aspire pins PostgreSQL 18.3)

The imageRegistry override applies to every un-qualified reference in both services and dependencies. Already-qualified references (those carrying a registry hostname) are never rewritten — they are pulled from their specified host as-is. When you specify any image: field on a dependency, the engine also clears any built-in registry default the provider might carry, preventing unintended double-prefixing. The guarantee is: an image: is used exactly as written (with the provider's registry default cleared), and imageRegistry is applied on top only when the image carries no registry hostname of its own.

2. Override dependency images with per-dependency image: field

For finer-grained control — when you need a specific version or a non-standard image for one dependency — use the image: field on individual dependencies:

environment:
  dependencies:
    orders-db:
      type: postgres
      image: nexus.corp.local:5000/platform/postgres:16-custom    # Explicit override

    events:
      type: kafka
      image: artifactory.mycompany.com/confluent/kafka:7.5.0       # Pulls from Artifactory

    cache:
      type: redis
      version: "7"                                                  # Uses Aspire's default: redis:7

The per-dependency image: field bypasses Aspire's provisioned default entirely. An image: carrying no tag or digest must be paired with a version: field; version: supplies the tag. If both image: (with a tag) and version: are set, that is rejected as ambiguous. A tagless image: without a sibling version: is rejected with a clear error: it would silently float on :latest, defeating the determinism invariant.

3. Enforce local-cache-only operation

For fully air-gapped or pre-warmed environments, use imagePullPolicy: Never to prevent unexpected outbound pulls:

environment:
  imagePullPolicy: Never    # All images must be pre-warmed locally; no external pulls allowed

  services:
    my-api:
      image: myco/myapi:v1.2.3

  dependencies:
    db:
      type: postgres
      version: "16"

If an image is not present locally, the topology will fail to start. Ensure all required images are pre-warmed on the host:

# Before running tests, pre-warm all images
docker pull myco/myapi:v1.2.3
docker pull postgres:16
docker pull redis:7

4. Built-in images and registry redirect scope

Some managed resources carry built-in container images. The scope of imageRegistry and per-dependency image: overrides varies by resource type:

kafka with schemaRegistry: true — vouchfx provisions a Confluent Schema Registry sidecar (pinned to confluentinc/cp-schema-registry:7.6.1). The sidecar image carries no embedded registry hostname, so imageRegistry does apply to it. If you set imageRegistry: artifactory.mycompany.com/docker-mirror, the sidecar will be pulled from artifactory.mycompany.com/docker-mirror/confluentinc/cp-schema-registry:7.6.1. There is no per-dependency image: override for this sidecar; authors must rely on imageRegistry redirection.

azureservicebus — vouchfx provisions two containers. The main emulator container (mcr.microsoft.com/azure-messaging/servicebus-emulator:1.1.2) and the SQL Server 2022 sidecar (mcr.microsoft.com/mssql/server@sha256:cf8f0ab2ebe61feec2eed2629061a07d247f791c2a8ddda74a4ac9f06a291718 (SQL Server 2022 CU27 GDR1)) both embed mcr.microsoft.com. Because their images are fully qualified, imageRegistry does not apply to either — the rule "already-qualified references are never rewritten" protects them both. The main emulator can be redirected per-dependency with an image: field naming your own registry (for example image: artifactory.mycompany.com/azure-messaging/servicebus-emulator:1.1.2); version: will not help here, because it only replaces the tag on the provider's built-in repository and cannot change the registry. The SQL sidecar cannot be overridden by any author-controlled means.

minio — vouchfx provisions a single container pinned to bitnamilegacy/minio:2025.7.23-debian-12-r5, a Bitnami build of MinIO 2025-07-23 on Docker Hub. That reference carries no registry hostname, so imageRegistry does apply to it, as it does to every other Docker Hub dependency. The built-in pin moved away from upstream MinIO's own image because upstream withdrew its images from Docker Hub and quay.io entirely; the Bitnami repository is a frozen archive ("Legacy Bitnami images (no longer updated)", last push 2025-08-19), so the build is older than the previous pin and receives no security fixes, and the maintained S3-compatible kind is s3 (#581), described below. A suite that must hold a specific digest can pin one per dependency with image: bitnamilegacy/minio@sha256:6dabb4a2088c9a79908de3bc05f4586c23ad2182c8908e7e3acbf61c1467fb20, which the engine uses as written, prefixed by imageRegistry when one is set, like any other unqualified image:.

If this dependency fails to pull, check which registry the run reached. With imageRegistry set, the pull goes to <imageRegistry>/bitnamilegacy/minio:2025.7.23-debian-12-r5, so a mirror that proxies only an allow-list of Docker Hub repositories must have bitnamilegacy/minio added to it. Without imageRegistry, the pull goes to Docker Hub directly and needs egress to it. Behind #533's exemption the same dependency used to reach quay.io and bypass the mirror; that no longer happens. To pull the build from a registry or repository of your own, override it per-dependency with image: artifactory.mycompany.com/bitnamilegacy/minio:2025.7.23-debian-12-r5. As above, version: will not help, because it only replaces the tag and cannot change the registry or repository. An image: swaps only the image, not the contract, so an override for minio must be a MinIO build that honours it (server /tmp/minio-data, MINIO_ROOT_USER/MINIO_ROOT_PASSWORD, readiness at /minio/health/cluster).

s3 — vouchfx provisions a single RustFS container (Apache-2.0). Its default is pinned by digest, rustfs/rustfs@sha256:8cc98017… on Docker Hub (the digest of RustFS 1.0.0), not by tag, because Aspire's image annotation holds a tag or a digest and never both. The repository name carries no registry hostname, so imageRegistry does apply to it: with it set, the pull goes to <imageRegistry>/rustfs/rustfs@sha256:8cc98017…, so a mirror that proxies only an allow-list of Docker Hub repositories must have rustfs/rustfs added to it. Without imageRegistry, the pull goes to Docker Hub directly. The mirror must serve the multi-arch index under the digest sha256:8cc98017…: a pull-through cache does this, and a manual copy should use docker buildx imagetools create, crane copy or skopeo copy --all, not pull/tag/push, which re-push one platform under a different digest, so the pinned reference then fails to pull and the run reports an environment error. A per-dependency version: replaces the digest with your tag, trading the content pin for a mutable tag (image: rustfs/rustfs@sha256:<digest> keeps a pin), and image: replaces the whole reference. An image: swaps only the image, not the contract, so an override for s3 must be a RustFS build that honours it (RUSTFS_ACCESS_KEY/RUSTFS_SECRET_KEY, port 9000, data directory /data, readiness at /health/ready).

Principle: An image: field overrides the main container only (never a sidecar); a registry-qualified value is used exactly as written, and an un-qualified value is prefixed by imageRegistry when one is set. imageRegistry reaches every un-qualified image (including the Kafka schema-registry sidecar) but not images with a registry hostname of their own. imagePullPolicy applies to all containers.

Container image pin maintenance

The hidden Azure Service Bus SQL sidecar, Vault and whoami Docker test fixtures, and the existing RustFS default use reviewed content digests. The release tags and full digests are recorded in scripts/ci/image-pins.json. Vault, whoami and RustFS use upstream multi-arch indexes covering Linux amd64 and arm64. SQL Server is an explicit exception: Microsoft publishes the selected image only for Linux amd64 and supports SQL Server containers on x86-64 Linux hosts. Pinning does not add ARM support.

An azureservicebus dependency's version: and image: select its main emulator; the internal SQL sidecar keeps its own pin. The existing s3 rule remains unchanged: an authored version: replaces its default digest with that tag; an authored image: replaces the whole reference. Other built-in dependency tags remain author-overridable.

Run python -B scripts/ci/check_image_pins.py from the repository root to compare each reviewed release tag with its pinned manifest and verify platform coverage. The Container image pin freshness workflow runs this read-only check weekly and on manual dispatch, with a 120-second acceptance deadline and socket-operation timeouts of at most 15 seconds. Responses arriving at or after the deadline are refused, but a blocked socket read is not guaranteed to stop at that exact moment; the hosted job has a separate five-minute limit. A mismatch or registry failure fails the audit; it never pulls containers or edits pins. A match means the selected release tag still identifies the pinned bytes; it does not prove that the release is the newest available or free of vulnerabilities.

Review upstream release and security notes at each engine release and when the audit fails. Submit deliberate pin updates in a PR: verify the upstream manifest digest and platform list, update the inventory, mapper or fixture references, expected identities and CI pre-pull list together, then run the source-contract tests and the affected Docker integration tests. Do not replace a multi-arch index with one platform's manifest or automatically accept a repointed tag. Digest identity is not signature or attestation verification.

Best practices for private registry operation

1. Use imageRegistry for wholesale redirects when you have a private mirror of all public images. This keeps scenarios concise.

2. Use per-dependency image: when you need per-resource control, or when not all images are mirrored (only some dependencies go to the internal registry).

3. Combine both: imageRegistry as a safety default, and per-dependency image: for exceptions:

   environment:
     imageRegistry: mirror.internal.corp    # Fallback for any un-qualified reference
     dependencies:
       orders-db:
         type: postgres                     # Will use mirror.internal.corp/library/postgres:18.3
       events:
         type: kafka
         image: kafka.example.com/cp-kafka:7.5  # Override: this one comes from a different host
   

4. Prefer digest pinning (@sha256:…) over tags in air-gapped environments; digests are byte-stable and prevent accidental pulls of newer versions if a tag is reused.

5. Document your registry strategy in your test suite README so teammates understand which images are sourced from where.

---

Discovery root does not exist (dotnet run path resolution gotcha)

Symptom:

Discovery root './tests/e2e' does not exist

when running:

dotnet run --project src/Cli/Vouchfx.Cli/Vouchfx.Cli.csproj -- run ./tests/e2e

What it means: This gotcha is specific to running the CLI from source via dotnet run --project. When you use dotnet run, it changes the working directory to the project directory (src/Cli/Vouchfx.Cli/), not your current directory. So a relative path like ./tests/e2e resolves relative to that project directory, not your repository root, and fails.

If you are using the packaged vouchfx CLI (installed via dotnet tool install), you will not encounter this issue — the tool runs from your current working directory, so relative paths work as expected.

Fix:

Use one of these approaches:

1. Use the packaged vouchfx CLI (recommended):

   # Install the latest stable CLI
   dotnet tool install --global vouchfx
   
   # Already installed? Upgrade instead
   dotnet tool update --global vouchfx
   
   # Then run from your current directory
   vouchfx run ./tests/e2e
   

The packaged tool runs from your current directory, so relative paths behave as expected and this gotcha never applies.

1. If you are contributing to vouchfx from source, build the CLI once and run the binary directly:

   # Build once
   dotnet build src/Cli/Vouchfx.Cli/Vouchfx.Cli.csproj -c Release
   
   # Run the built binary
   src/Cli/Vouchfx.Cli/bin/Release/net8.0/vouchfx run ./tests/e2e
   

   (On Windows: src\Cli\Vouchfx.Cli\bin\Release\net8.0\vouchfx.exe run .\tests\e2e)

The binary runs from your current directory, so relative paths work as expected.

1. Pass an absolute path to dotnet run:

   dotnet run --project src/Cli/Vouchfx.Cli/Vouchfx.Cli.csproj -- run $(pwd)/tests/e2e
   

   (On Windows: dotnet run --project src/Cli/Vouchfx.Cli/Vouchfx.Cli.csproj -- run $((Get-Location).Path)/tests/e2e)

Best practice: For running container-backed vouchfx test suites, use the packaged CLI with the supported .NET 8 runtime, Docker running Linux containers and its version-specific DCP prerequisites. Published 1.0.0 bundles matching DCP tools without a source checkout, Aspire workload or warmed NuGet cache. Legacy rc.6 requires externally provisioned matching tools. The packaged CLI runs from your current directory without path-resolution workarounds. If you are working on vouchfx itself from source, build the CLI once and run the binary directly — it is faster (skips compilation) and avoids dotnet run quirks altogether.

---

Script body, document, and YAML expansion limits

Symptom:

script.csharp: 'code' size 72000 characters exceeds the 65536-character limit (a plain resource bound, not a security control — see this file's header comment); reduce its size or split the script.

or

File size 1200000 bytes exceeds the 1048576-byte (1 MiB) limit for a single *.e2e.yaml document (a guard against pathological input); split the suite into smaller files.

or

Failed to parse YAML: The YAML document expands to more than 16,777,216 characters (16 Mi) of JSON during schema conversion…

What it means:

The engine enforces three resource-limit bounds before compilation: a maximum of 64 KiB per script.csharp step body (inline code or referenced file:), 1 MiB per .e2e.yaml document, and 16 Mi characters (16,777,216) of JSON output when the YAML is converted during validation. These are sanity bounds to prevent accidentally passing pathologically large files to the compiler, not a defence against deliberate crash or hang attempts — which can occur well under these sizes (e.g. a ~100-character nested string interpolation can hang the parse). The third bound is triggered by YAML anchors and aliases that expand to many copies of repeated content.

When a limit is exceeded, the scenario is marked Inconclusive on run and invalid on validate, exiting 4 on both — with or without --fail-on-inconclusive. The document cap is a parse failure (#425) and the script-body cap refuses before any topology is built (#369), so neither can report the run as clean. Validation completes normally and names the specific limit — it is not a crash.

Fix:

1. Script body too large? Split a large script.csharp body across multiple steps:

   - id: validate-part-1
     type: script.csharp
     code: |
       // First portion of logic
   
   - id: validate-part-2
     type: script.csharp
     code: |
       // Next portion of logic
   

2. Document too large? Split into separate .e2e.yaml files:

   # Before: one large scenario file (1.5 MiB)
   # After: split by concern
   create-tests.e2e.yaml       # Creation tests (~200 KiB)
   fulfillment-tests.e2e.yaml  # Fulfillment tests (~300 KiB)
   audit-tests.e2e.yaml        # Audit trail tests (~250 KiB)
   

3. YAML expands too large during validation? Reduce YAML aliases and anchors, or split the suite. Anchors and aliases (the & and * syntax) expand to repeat their target content, and deeply nested or frequently reused anchors can make the expanded JSON exceed 16 Mi characters during validation.

---

Understanding the four verdicts

vouchfx distinguishes four outcomes (see docs/01 §12.1 for the full taxonomy):

Verdict	Meaning	Example	Default CI exit code
Pass	All assertions passed.	A test runs end-to-end and all steps succeed.	0 (success)
Fail	An assertion failed — a genuine product defect.	expect: { status: 200 } but the API returned 500.	1 (always breaks CI)
EnvironmentError	Infrastructure problem, not a product defect.	Docker daemon unreachable, image pull fails, seed SQL fails.	0 by default; 3 with --fail-on-env-error; a security refusal follows the exit-code policy
Inconclusive	The engine could not decide; the assertion may pass if retried.	A RETRY step's polling window expires; a capture expression fails to match.	0 by default; 4 with --fail-on-inconclusive; refusals follow the exit-code policy

Why the distinction?

In microservices, infrastructure is often brittle. A test might fail not because the code is wrong, but because Docker is slow or a database is down. The vouchfx verdict taxonomy lets your CI system handle each case independently:

• Fail (1) — Your code broke. Fix it.
• EnvironmentError (0 or 3) — The infrastructure is in trouble. Page on-call or escalate to DevOps.
• Inconclusive (0 or 4) — The engine could not decide; maybe timing was just off. Investigate or re-run.

An aggregate Fail exits 1. Ordinary unrelated or unattributed infrastructure errors retain opt-in gating. A typed readiness, discovery or server-material access failure of the declared secured target itself prevents confirmation and exits non-zero without either flag. EnvironmentError ranks above Fail: a failed assertion beside that secured-target failure therefore aggregates to EnvironmentError and exits 3. See CI integration for authoring and provider-refusal rules.

Refusals can break CI without the gating flags. A security declaration raises the default exit when it is refused for its contents, fails confirmation, or its own named readiness, discovery or server-material access fails before confirmation. An unrelated or unattributed topology failure remains opt-in; the presence of a security block does not make every infrastructure failure unconditional. The run prints a security notice on stdout when this rule applies.

Parse, authoring, no-execution and provider/engine refusal rules are listed in the exit-code policy. Provider guards cover Bind, Validate, Resources, HostResources, Emit, CompileReferenceAssemblies and fragment assembly. Read the diagnostic before assigning a provider bug: a guard can also expose an engine dispatch or host filesystem failure.

An aggregate Fail exits 1. The existing aggregate ranking places EnvironmentError above Fail, so a completed assertion failure beside an unconfirmed secured target's own readiness failure aggregates to EnvironmentError and exits 3. A topology failure with no attributable security failure or other refusal still exits 0 by default, or 3 with --fail-on-env-error. See CI integration for mixed-result precedence.

Opt into stricter gating with flags:

# Fail breaks CI; environment errors do not (default)
vouchfx run ./tests

# Also break CI on environment errors
vouchfx run ./tests --fail-on-env-error

# Also break CI on inconclusive results
vouchfx run ./tests --fail-on-inconclusive

# Both
vouchfx run ./tests --fail-on-env-error --fail-on-inconclusive

---

Secret leakage in exception messages

Symptom: A script.csharp step throws an exception with a secret value in the message. The exception message is recorded as an observation in the --events JSON Lines output, where it becomes visible.

Example:

- id: validate-token
  type: script.csharp
  code: |
    var token = Vars.Secrets.Resolve("vault/api-token").Reveal();
    if (!token.StartsWith("sk_test_"))
      throw new Exception($"Invalid token: {token}");  // DANGER: reveals the token

What it means: The --events JSON Lines stream passes through the same scrubbing as the terminal output and HTML report (both redact secret values and substitute declared paths). If a script throws an exception with a revealed secret in its message, that message is redacted before reaching the event stream, just as it is on the terminal.

Fix:

1. Never embed resolved secret values in exception messages. Use the reference path or a generic error message instead:

   // Good: mentions the reference, not the resolved value
   throw new Exception("Token validation failed (source: vault/api-token)");
   
   // Also good: no secret mention at all
   if (!IsValidToken(token))
     throw new Exception("Invalid token format");
   

2. The only deliberate escape hatch is SecretString.Reveal(). Call it only at the moment you inject the value into a sink (e.g. an HTTP Authorization header), never write the revealed value back into Vars or any logged/serialised structure:

   var token = Vars.Secrets.Resolve("vault/api-token").Reveal();
   // Use token here at the injection point only; never store or log the revealed value
   if (!IsValidToken(token))
     throw new Exception("Token validation failed");  // No secret in the message
   

3. Understand the three tiers of secret protection (see docs/01 §17):

4. Tier 1: No bake into IL — The secret value is never compiled into the C# source or IL (verified by SecretResolutionPipelineTests).
5. Tier 2: Redaction in output — Terminal output, HTML report, JUnit XML, and --events JSON Lines all redact secret values and substitute declared paths (values appear as ***REDACTED***; resolved paths appear as the author's original declared text).
6. Tier 3: Author discipline — Beyond the engine's automated redaction, authors must still avoid embedding secrets in exception messages because the scrub cannot catch deliberately transformed values (base64, HMAC, substrings).

Best practice: Treat exception messages as user-visible; never embed secrets in them. The reproducibility envelope records the reference hash, not the value — use that for reproducibility without embedding secrets.

---

Build fails with warnings-as-errors

Symptom:

build.csproj : error : Treating warnings as errors.
CSC5001: warning CS8360: [details...] cannot have...

What it means: The vouchfx codebase is compiled with /WarnAsError enabled (zero-warning policy). Any compiler warning (including #nullable, unused variable, etc.) is treated as a build error.

Fix:

1. Address the root cause: Read the warning message and fix it. Common examples:
2. Unused using statement → remove it.
3. Null-reference warning → add a null check or ! operator.

4. Unused variable → remove it or prefix with _.

5. If the warning is spurious or unavoidable: Suppress it locally with #pragma:

   #pragma warning disable CS8360
   // Your code here
   #pragma warning restore CS8360
   

6. Run the format gate locally before pushing:

   dotnet format --verify-no-changes
   

   This catches formatting issues (and indirectly, some warnings) early.

Why the strict policy? Zero-warning builds improve code quality and make intentional (narrowly-scoped) suppressions stand out in code review.

---

Aspire topology timeouts or hangs

Symptom: Tests hang or timeout waiting for the topology to start. The Aspire dashboard (if enabled) shows resources stuck in a transitional state.

What it means: A service or dependency failed to start or became unhealthy. Common causes:

1. A service depends on another service that is not declared. Aspire cannot auto-discover undeclared dependencies.
2. A healthcheck endpoint is broken. vouchfx health-gates ports; if the port is open but the healthcheck fails, Aspire retries indefinitely.
3. A container image does not exist or cannot be pulled. docker pull fails silently in some Aspire configurations.

Fix:

1. Run with the --events flag to inspect observations to see which resource is stuck:

   vouchfx run ./tests --events ./events.jsonl
   

   Then examine events.jsonl for step-attempt records showing stuck health-check attempts or failed container starts.

2. Verify all dependencies are declared in environment.services and environment.dependencies. If a service tries to connect to a database not in the environment, the topology cannot satisfy it.

3. Verify the healthcheck endpoint is fast and works. Test it manually:

   docker run -d -p 8080:8080 myco/myservice:latest
   curl http://localhost:8080/health
   

   If the healthcheck endpoint hangs or fails, the container will be marked unhealthy.

4. Check image availability:

   docker pull myco/myservice:latest
   

   If the pull fails, the topology cannot start.

---

Steps run but assertions fail

Symptom: A step runs (e.g., an HTTP request succeeds with status 200) but the assertion fails:

Assertion failed: expected status 201, got 200

What it means: The step executed successfully, but the response did not match the expected conditions. This is a Fail verdict — a genuine product defect, not an infrastructure problem.

Fix:

1. Review the assertion. Is it correct for the current test? http.rest asserts the status code and, through expect.json and expect.bodyContains, the response body. A failing body assertion's observation names the first one that did not hold — its JSONPath, a reason such as mismatch, missing or notJson, and your expected value, each cut to 256 characters with a trailing … if longer — with the JSON kind of what was found, never the response's own text. Remember that expect.json compares text: 2 does not match 2.0, and an equality path must select exactly one node.

   expect:
     status: 201  # Should this be 200?
     json:
       "$.status": PENDING  # Or has the service already moved it on?
   

2. Examine the actual response. Run the step in isolation (if possible) and log the full response:

   - id: debug-response
     type: http.rest
     target: api
     method: GET
     path: /data
     # No expect block — just capture the response
     capture:
       response_body: "$"
   
   - id: log-it
     type: script.csharp
     code: |
       var resp = (string)Vars["response_body"];
       System.Console.WriteLine("Response: " + resp);
   

3. Check for state threading issues. If an earlier step's capture is empty, a placeholder substitution may fail:

   - id: create
     type: http.rest
     target: api
     method: POST
     path: /resources
     capture:
       resource_id: "$.id"  # Did this JSONPath actually match?
   
   - id: fetch
     type: http.rest
     target: api
     method: GET
     path: "/resources/{resource_id}"  # Is {resource_id} really set?
   

4. Re-run the scenario with --events to capture raw observations:

   vouchfx run ./tests --events ./events.jsonl
   

   The JSON Lines file contains every step observation; examine it for the actual response.

---

Capture fails or placeholder is empty

Symptom:

Capture failed: JSONPath '$.missing_field' did not match the response

or a placeholder resolves to empty/null in a later step.

What it means: A capture expression (JSONPath or XPath) did not match the step result, or a placeholder references a non-existent variable.

Fix:

1. Verify the JSONPath is correct. Use a JSONPath tester (e.g., jsonpath.com) to test the expression against your response:

   // Response
   { "user": { "id": 123, "name": "Alice" } }
   
   // Correct path
   $.user.id  // → 123
   
   // Incorrect path
   $.id       // → null (user is nested)
   

2. Check the response structure. The actual response may differ from what you expected:

   capture:
     id: "$.id"  # Response has { "userId": 123 }, not { "id": 123 }
   

3. Guard optional data explicitly before capturing it. A missing JSONPath result is Inconclusive; captures have no implicit default:

   # A missing value does not silently receive a default.
   # Check optional response fields in your assertion/script logic first:
   capture:
     error_code: "$.error.code"  # If 'error' is missing, the capture fails
     # Instead, check the response structure first in a script:
   

4. Keep the capture verdict distinct from continuation. continueOnFailure does not turn a missing capture into Pass:

   - id: get-optional-data
     type: http.rest
     target: api
     method: GET
     path: /data
     expect:
       status: 200
     capture:
       optional_field: "$.optional"
     continueOnFailure: true  # A capture miss still reports Inconclusive
   

5. Verify variable names match. Placeholders are case-sensitive:

   capture:
     UserId: "$.id"  # Captured as "UserId"
   
   # Later step
   path: "/users/{userid}"  # ERROR: {userid} ≠ {UserId}
   

---

Kafka messages not consumed (ordering or timing)

For a managed broker with security:, start with the tested PEM TLS/mTLS recipe. An invalid broker PEM can pass metadata-only validation and then fail readiness; check the declared server artefacts and matching PEM type/location settings. Do not override the managed listener mechanics or replace Aspire's allocated host address with a guessed port. The selected listener is secured; Schema Registry and container connection references use the other plaintext listener.

The 1.0.0 assertion consumer manually assigns every actual topic partition at its earliest retained offset. It does not wait for a consumer-group rebalance. Metadata discovery and the record drain share a one-second attempt window; an empty partition does not stop the others. Use verifyMode: RETRY when the matching record may arrive later. The step timeout controls the overall retry budget, not a wider individual drain.

Your application's own subscribing consumer may need group-startup tuning. That is separate from this assertion drain; KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: "0" is not a prerequisite for the vouchfx assertion. See Compose broker bindings for host-reachable advertised listeners and native readiness.

Symptom: An mq-expect.kafka step fails to find a message that was published earlier, even with verifyMode: RETRY.

What it means: Common causes:

1. Message expired or was evicted before the step ran. Each assertion attempt explicitly assigns the topic's actual partitions at their earliest retained offsets, so a record published before the step starts remains eligible. A record already removed by retention cannot be recovered; check topic retention against the publish-to-expect interval.
2. Topic does not exist. The message was published to a different topic.
3. Key or match criteria are too strict. The message exists but does not match the filter.
4. Timing issue. The system under test has not yet published the matching record. Suite steps run in order; use verifyMode: RETRY for asynchronous application work.

Fix:

1. Ensure the order: publish → expect. In your steps, publish first, then expect:

   - id: publish-order-event
     type: mq-publish.kafka
     target: kafka
     topic: orders.created
     payload: |
       { "orderId": "123", "status": "new" }
     key: "123"
   
   # Later step (runs after publish completes)
   - id: expect-order-event
     type: mq-expect.kafka
     target: kafka
     topic: orders.created
     match:
       key: "123"
     verifyMode: RETRY
     timeout: 10s
   

2. Use a consistent topic name. Verify the publish and expect steps reference the same topic:

   - id: publish
     type: mq-publish.kafka
     topic: order.created  # All lowercase
   
   - id: expect
     type: mq-expect.kafka
     topic: order.created  # Must match exactly
   

3. Relax the match criteria temporarily to debug. Remove the key or other filters to see if the message exists:

   match:
     # Removed strict key filter for debugging
     payloadContains: "orderId"
   

4. Use verifyMode: RETRY with a reasonable timeout (e.g., 10–30 seconds):

   - id: expect-with-polling
     type: mq-expect.kafka
     target: kafka
     topic: orders.created
     match:
       key: "123"
     verifyMode: RETRY
     timeout: 10s  # Wait up to 10 seconds, polling with backoff
   

5. Check Kafka broker health. If the broker is slow or unhealthy, messages may be delayed:

   docker logs <kafka-container-id>
   

6. Understand Kafka's offset management. Each attempt explicitly assigns every topic partition at its earliest retained offset. Discovery and consumption share the bounded attempt window; EOF on one partition does not end the others. Ensure the broker's advertised listener is reachable from the CLI host, the declared security material matches, and the publish step precedes the expect step. Retained data remains eligible, but a bounded attempt may need another RETRY attempt to find a match.

---

RabbitMQ: message not in queue (ownership, durability, publish-after-declare)

Symptom: An mq-expect.rabbitmq step asserts on a queue but the message is not found, even though the publish step ran first.

What it means: Common causes:

1. Queue does not exist. RabbitMQ routing requires the queue to exist before a message can be routed to it.
2. Durability mismatch. The queue was declared as transient (auto-delete) in a previous test and was deleted when emptied.
3. Wrong routing key. The routing key used in the publish step does not match the queue name or binding.
4. Message was already consumed. Another consumer drained the queue before the assertion ran.

Fix:

1. Declare the queue explicitly in the environment. If your SUT declares the queue at startup, ensure the HTTP trigger step runs before the assertion:

   steps:
     # Step 1: Trigger SUT to declare the queue
     - id: initialize-service
       type: http.rest
       target: rabbitmq-service
       method: POST
       path: /init
       expect:
         status: 200
   
     # Step 2: Now the queue exists; publish a message
     - id: publish-message
       type: mq-publish.rabbitmq
       target: events-rmq
       routingKey: orders-notifications
       payload: '{"orderId":"123"}'
   
     # Step 3: Assert the message is in the queue
     - id: assert-message
       type: mq-expect.rabbitmq
       target: events-rmq
       queue: orders-notifications
       match:
         payloadContains: "123"
       verifyMode: RETRY
       timeout: 10s
   

2. Use durable queues. In your SUT or topology setup, declare queues with the durable flag set to true:

   // Example: declaring a durable queue in RabbitMQ client
   channel.QueueDeclare(
       queue: "orders-notifications",
       durable: true,
       exclusive: false,
       autoDelete: false
   );
   

3. Match the routing key to the queue name. When using the default exchange, the routing key must match the queue name exactly:

   - id: publish
     type: mq-publish.rabbitmq
     routingKey: orders-notifications  # Routes to this queue name
   
   - id: assert
     type: mq-expect.rabbitmq
     queue: orders-notifications  # Same as routing key
   

4. Use verifyMode: RETRY to wait for eventual consistency:

   - id: assert-with-retry
     type: mq-expect.rabbitmq
     target: events-rmq
     queue: orders-notifications
     match:
       payloadContains: "123"
     verifyMode: RETRY
     timeout: 15s
   

---

NATS: Inconclusive verdict on mq-expect.nats (stream created after publish — lost message)

Symptom: An mq-expect.nats step times out with an Inconclusive verdict, but the message was published in a preceding step.

What it means: NATS JetStream messages are only retained if the stream exists before they are published. If you publish to a subject before the stream is created, the message is lost and cannot be recovered. The step then polls indefinitely and times out.

External-producer scenario: the system under test publishes before a matching JetStream stream exists, then a later expect step creates the stream after that message has already been lost. Configure the stream and its subjects before triggering the system under test.

The built-in mq-publish.nats provider creates the requested stream before publishing. A normal built-in publish → expect sequence does not require a deliberately failing warm-up assertion.

Fix:

1. Create the stream before an external producer sends messages. Declare it in the application or broker setup, with subjects matching the publication. When using the built-in publisher, use the same explicit stream and subject in the publish and expect steps:

   steps:
     - id: publish-order-event
       type: mq-publish.nats
       target: nats-broker
       subject: orders.created
       stream: ORDERS_CREATED
       payload: '{"orderId":"123"}'
   
     - id: assert-order-event
       type: mq-expect.nats
       target: nats-broker
       subject: orders.created
       stream: ORDERS_CREATED
       match:
         payloadContains: "123"
       verifyMode: RETRY
       timeout: 10s
   

2. Use explicit stream names to avoid derivation ambiguity. When two subjects might derive to the same stream name, use explicit names:

   - id: publish
     type: mq-publish.nats
     target: nats-broker
     subject: orders.created
     stream: ORDERS_CREATED  # Explicit name prevents collisions
   
   - id: assert
     type: mq-expect.nats
     target: nats-broker
     subject: orders.created
     stream: ORDERS_CREATED  # Must match the publish step
   

3. Understand stream name derivation. Subject names are converted to stream names by uppercasing and replacing non-alphanumeric characters (except -) with underscores. For example:

4. orders.created → ORDERS_CREATED
5. orders_created → ORDERS_CREATED (collision!)
6. orders-created → ORDERS-CREATED (different from orders.created)

When in doubt, use explicit stream names in both publish and expect steps.

1. Use separate NATS dependencies per scenario. If you have multiple scenarios asserting on the same subject, give each scenario its own nats dependency to avoid cross-scenario message bleed:

   # Scenario 1
   environment:
     dependencies:
       nats-scenario-1:  # Separate instance
         type: nats
   
   # Scenario 2
   environment:
     dependencies:
       nats-scenario-2:  # Separate instance
         type: nats
   

---

Azure Service Bus: entity not declared in dependency config → EnvironmentError

Symptom:

RunSuiteAsync: topology failed to start - queue 'my-queue' not found on service-bus dependency

What it means: Azure Service Bus Emulator (and real Azure Service Bus) requires queue and topic declarations to be made explicit in the test's environment.dependencies section. Queues and topics that are not declared will not be created, and messages sent to them will fail with an EnvironmentError.

Fix:

1. Declare all queues and topics in the environment. List them explicitly under the azureservicebus dependency:

   environment:
     dependencies:
       service-bus:
         type: azureservicebus
         queues:
           - order-commands
           - order-notifications
         topics:
           - name: order-events
             subscriptions:
               - fulfillment-sub
               - billing-sub
           - name: shipment-events
             subscriptions:
               - tracking-sub
   

2. Match queue and topic names exactly. Ensure the mq-publish and mq-expect steps reference the declared names:

   steps:
     - id: publish
       type: mq-publish.azureservicebus
       target: service-bus
       queue: order-commands  # Must be declared above
   
     - id: assert
       type: mq-expect.azureservicebus
       target: service-bus
       queue: order-commands  # Same name
   

3. For topic subscriptions, declare both the topic and subscription. A subscription cannot be created without its parent topic:

   environment:
     dependencies:
       service-bus:
         type: azureservicebus
         topics:
           - name: order-events
             subscriptions:
               - fulfillment-sub  # This subscription must have a parent topic
   

4. Understand the entity declaration syntax.

5. Queues are simple strings: queues: [queue1, queue2, ...]
6. Topics are objects with a name and optional subscriptions array: topics: [{name: topic1, subscriptions: [sub1, sub2]}]

---

Redis: AuthenticationError from the SUT (missing password credential)

Symptom: Your system under test fails with an authentication error when connecting to Redis:

AuthenticationError: WRONGPASS invalid username-password pair

or

ERR invalid password

What it means: The managed Redis dependency (provided by Aspire/Testcontainers) has authentication enabled, and your SUT is either: 1. Not providing the password at all. 2. Using the wrong password or username.

Fix:

1. Inject the Redis password via environment variable. The engine stages the Redis connection string (including password) as conn::<target>. Inject it into your SUT:

   environment:
     services:
       my-service:
         image: myco/my-service:latest
         env:
           REDIS_CONNECTION_STRING: ${conn:cache}
   

Your SUT should then parse this connection string and use it to connect.

1. If your SUT requires host, port, and password separately, extract them:

   environment:
     services:
       my-service:
         image: myco/my-service:latest
         env:
           REDIS_HOST: ${conn:cache.host}
           REDIS_PORT: ${conn:cache.port}
           REDIS_PASSWORD: ${conn:cache.password}
   

2. For .NET applications using StackExchange.Redis, construct the connection options from the credentials:

   string connString = Environment.GetEnvironmentVariable("REDIS_CONNECTION_STRING");
   var options = ConfigurationOptions.Parse(connString);
   var redis = ConnectionMultiplexer.Connect(options);
   

3. If the error persists, verify the Redis container is healthy:

   docker logs <redis-container-id>
   redis-cli -h localhost -p 6379 -a <password> ping
   

---

SQL Server: slow cold start; EnvironmentError on first pull

Symptom:

RunSuiteAsync: topology failed to start - HealthGate timeout of 00:00:20 on resource 'sqlserver'

The SQL Server container takes a long time to start for the first time.

What it means: SQL Server is a large image (multiple gigabytes) and has a lengthy initialisation sequence. On the first run, Docker must pull the image from the registry, which can exceed the Aspire/DCP 20-second health check window.

Fix:

1. Pre-warm the SQL Server image locally before running the suite (most effective):

   # Managed sqlserver dependency: Aspire default (unless version is overridden).
   docker pull mcr.microsoft.com/mssql/server:2022-latest
   # Azure Service Bus SQL sidecar: engine digest pin (2022-CU27-GDR1-ubuntu-22.04).
   docker pull mcr.microsoft.com/mssql/server@sha256:cf8f0ab2ebe61feec2eed2629061a07d247f791c2a8ddda74a4ac9f06a291718
   

   Pull the reference your topology uses: the managed sqlserver dependency uses Aspire's default tag above unless you set version:; the hidden Azure Service Bus SQL sidecar uses the exact digest from scripts/ci/image-pins.json. Pulling only the default tag does not warm the pinned sidecar. Once the matching image is cached, subsequent startups avoid the cold pull.

2. In CI, use the prewarm-images workflow input (GitHub Actions):

   vouchfx-e2e:
     uses: tomas-rampas/vouchfx/.github/workflows/vouchfx-run.yml@<commit-sha>
     with:
       scenario-path: ./tests/e2e
       prewarm-images: |
         mcr.microsoft.com/mssql/server:2022-latest
         mcr.microsoft.com/mssql/server@sha256:cf8f0ab2ebe61feec2eed2629061a07d247f791c2a8ddda74a4ac9f06a291718
   

3. Ensure sufficient disk space and memory. SQL Server requires at least 2 GB RAM and adequate disk space. If your Docker daemon is resource-constrained, increase limits.

4. Understand the startup sequence. SQL Server's health check waits for the database engine to be ready, which involves filesystem initialisation. This is non-configurable; the 20-second Aspire window is a hard limit.

---

SMTP/Mailpit: mail not found (wrong target key — HTTP API vs. SMTP endpoint)

Symptom: Your system sends an email via SMTP, but the mail-expect.smtp step finds no messages.

What it means: Common causes:

1. Your SUT is not connecting to Mailpit. The SMTP endpoint is not injected or the SUT is ignoring it.
2. Wrong environment variable names. The SUT expects different variable names than what the test provides.
3. Mailpit HTTP API is not reachable. The step is querying the wrong endpoint.

Fix:

1. Inject both the SMTP host and port separately. Mailpit exposes two endpoints:
2. HTTP API for querying: conn::mailpit (e.g., http://localhost:8025)
3. SMTP server for sending: svc::mailpit-smtp with separate host and port

Ensure your SUT gets both:

environment:
  services:
    my-app:
      image: myco/my-app:latest
      env:
        SMTP_HOST: ${conn:mail.host}
        SMTP_PORT: ${conn:mail.port}

1. Verify the SUT is sending email. Add a debug HTTP step before the mail assertion to confirm the SUT is processing the request:

   - id: send-email-request
     type: http.rest
     target: my-app
     method: POST
     path: /send-welcome
     body:
       to: alice@example.com
       subject: "Welcome"
     expect:
       status: 202
   

2. Use verifyMode: RETRY to wait for SMTP delivery:

   - id: assert-email
     type: mail-expect.smtp
     target: mail
     expect:
       count: 1
       match:
         to: alice@example.com
         subject-contains: "Welcome"
     verifyMode: RETRY
     timeout: 30s
   

3. Check Mailpit is running. Verify the container is healthy:

   curl http://localhost:8025/api/v1/messages  # Should return JSON
   

4. Understand match semantics. All criteria (to, subject-contains, body-contains) are AND-conjunctive:

   match:
     to: alice@example.com
     subject-contains: "Welcome"
     body-contains: "Thank you"
   # Message must match ALL three criteria
   

If you are not finding a message, relax the criteria one by one to isolate the mismatch:

# First, check if ANY email exists to this address
match:
  to: alice@example.com

# Then add subject filtering
match:
  to: alice@example.com
  subject-contains: "Welcome"

# Finally, add body filtering
match:
  to: alice@example.com
  subject-contains: "Welcome"
  body-contains: "Thank you"

---

Cross-scenario state bleed on non-reset stores (DynamoDB, MinIO, S3)

Symptom: When you run multiple scenarios sequentially on the same topology, data from one scenario is visible in the next scenario. This happens with datastores that are not automatically reset (DynamoDB, MinIO and S3).

What it means: DynamoDB, MinIO and S3 (s3) are not cleared between sequential scenarios — any writes one scenario makes persist into the next. This is a deliberate limitation (out of scope). Other datastores (PostgreSQL, SQL Server, MySQL, MongoDB, Redis, Elasticsearch) are automatically reset between scenarios.

Solutions:

1. Run scenarios in parallel (preferred). Parallel execution isolates each scenario in its own topology, avoiding any cross-scenario state:

   # Run all scenarios in parallel (each gets its own containers); N must be >= 1
   vouchfx run tests/e2e --parallel 4
   

2. Add explicit cleanup steps. As the first step of each scenario, explicitly clean the datastore:

   steps:
     # Scenario 2: start fresh
     - id: cleanup-dynamodb
       type: script.csharp
       description: Clear DynamoDB tables for this scenario.
       code: |
         var config = new Amazon.DynamoDBv2.AmazonDynamoDBConfig { ServiceURL = (string)Vars["conn::dynamo"] };
         var client = new Amazon.DynamoDBv2.AmazonDynamoDBClient(config);
         try
         {
             // Delete items you wrote in the prior scenario (application-specific logic)
         }
         finally { client.Dispose(); }
   

Note: script.csharp contributes no compile references of its own — the AWS SDK types above compile only because a step from a provider that contributes them (such as db-assert.dynamodb) appears somewhere in the same scenario. A scenario whose only DynamoDB interaction is this cleanup script will fail to compile; include at least one db-assert.dynamodb step, or perform the cleanup through your SUT's own API instead.

1. Use scenario-scoped keys. Prefix your test data with a scenario identifier, then delete by prefix pattern rather than table-wide flush.

---

State reset failed (environment error naming a dependency)

Symptom: A scenario fails during the reset phase (between scenarios in a sequential suite) with an environment error message like:

state reset (postgres) reset failed for dependency 'orders-db': …

What it means: The engine attempted to clear a dependency's state between scenarios (the reset runs after each scenario completes in a sequential multi-scenario suite), but the reset operation failed. This is an environment error — a failure of the test infrastructure, not a failure of the system under test. Possible causes:

1. Store became unhealthy mid-suite — a container crashed, lost network connectivity, or hit a resource limit. Check Docker logs: docker logs <container-id>.
2. SQL Server with exotic schemas — temporal (system-versioned) tables are handled, but Respawn may fail on other unusual patterns (cross-database foreign keys, graph tables); consult Respawn's documented limitations.
3. MongoDB capped or time-series collections — these collections reject the document deletion the reset performs; remove them from the seeded data or use standard collections.
4. Elasticsearch reporting per-document failures — a delete-by-query partially failed; check Elasticsearch logs and ensure the cluster has sufficient resources.
5. Connection/authentication lost — the test infrastructure's credentials or network path to the store changed between scenarios.

Solutions:

1. Check the event stream for the full reset error message — it names the dependency and the specific failure stage (init, create, or reset):

   vouchfx run tests/my-scenario.e2e.yaml --events my-events.jsonl
   # Read the failure details from the environment-error event
   

2. Inspect container health — if a Docker container reset failed:

   docker ps | grep <dependency-name>
   docker logs <container-id>
   

3. For MongoDB, remove capped and time-series collections from the test data or switch to standard collections.

4. For SQL Server, temporal tables are handled automatically; if the reset still fails, look for other exotic schema patterns (cross-database foreign keys, graph tables) and consult Respawn's documented limitations.

5. For Elasticsearch, ensure the cluster is healthy and has sufficient resources; check the cluster health status:

   curl http://localhost:9200/_cluster/health
   

---

Validation errors at authoring time

Unknown step type (vouchfx validate or pre-compilation)

Symptom:

FAIL  my-test.e2e.yaml
    [Schema] (line 45) unknown step type 'db-assert.oracle' - not a registered provider (expected <family>.<provider>, e.g. 'db-assert.postgres').

What it means: A step's type field is well-formed (matches the <family>.<provider> pattern) but does not match any registered provider. This is caught early by vouchfx validate (compile-level validation without Docker) with a precise line number and stage, so you can fix the provider name before attempting to run the suite. Note: malformed types like a bare postgres or db_assert.postgres (with underscore) are rejected by the schema's pattern check instead, reported as separate [Schema] errors — the "unknown step type" message specifically catches well-formed-but-unregistered types.

Fix:

1. Check the step type against the registered providers. Consult the Language Reference or run vouchfx list to see all available step types:

   vouchfx list
   

2. Correct the type field to a registered provider. Ensure it follows the <family>.<provider> naming convention and matches one of the Core providers:

   # Schema error: malformed (missing family)
   - id: my-step
     type: postgres
   
   # Schema error: malformed (wrong separator)
   - id: my-step
     type: db_assert.postgres
   
   # Unknown-type error (well-formed but not a Core provider)
   - id: my-step
     type: db-assert.oracle
   
   # Correct (matches a registered Core provider)
   - id: my-step
     type: db-assert.postgres
   

3. Use vouchfx validate before running. It catches both malformed types (schema pattern errors) and well-formed-but-unregistered types (unknown-step-type errors), plus missing required fields, without needing Docker:

   vouchfx validate ./tests/e2e
   

---

Git selection diagnostics

When using --changed-since, the CLI invokes git to determine which scenarios have changed. Three distinct error messages indicate different failure modes:

Symptom 1:

Could not run git for <operation>. Is git installed and on PATH?

What it means: The bounded CLI search found no eligible file called git (or git.exe on Windows). Either git is not installed, or its installation directory is not in the PATH environment variable.

Fix:

1. Verify git is installed:

   git --version
   

   If this command fails, install git for your platform.

2. Add git to PATH:

3. Windows: Git for Windows installer typically adds git.exe to PATH automatically. If you installed git manually or to a custom location, add its bin directory to your PATH. Note: The selection looks only for git.exe, not .cmd or .bat shims. If your PATH contains only a git shim (e.g. from Chocolatey or another package manager), you must install the full Git for Windows package (which includes git.exe) or locate a real git.exe and add that directory to PATH before the shim directory.
4. macOS: If you installed git via Homebrew or Xcode, it is already on PATH. If installed manually, verify the installation directory is in PATH:

   which git
   

5. Linux: Install via your package manager (e.g. sudo apt-get install git), which adds it to PATH.

6. Verify PATH from vouchfx: The selection searches PATH in the order it appears in your environment. Print your current PATH:

   # Windows (PowerShell)
   $env:PATH
   
   # POSIX
   echo $PATH
   

   and confirm that the directory containing your git executable appears in this list.

---

Symptom 2:

Could not start git for <operation>. A git executable was found on PATH, but the operating system refused to start it. Is the selected git on PATH a valid executable this user may run?

What it means: A selected candidate could not start for an error outside the supported load-recovery cases, or the pinned executable failed during diff or status. No alternative is used after a binary has started. Verify that the discovery directory exists and that your git installation is accessible and complete. A .cmd or .bat shim is never a candidate.

Symptom 3:

Could not start git for the change-set computation. The bounded PATH search found candidates, but the operating system refused to load them. Install a valid git executable this user may run and place its directory earlier on PATH.

What it means: Every attempted candidate failed before execution with a supported OS load error. Selection examines at most 220 complete PATH entries and 32,768 UTF-16 code units, and attempts at most 32 distinct candidates. It skips empty and relative entries without resolving them against the current directory. It does not launch a shell to interpret a text file.

Fix:

1. Install a complete git distribution for your platform. On Windows, use a real git.exe; on POSIX, check execute permission and any required interpreter or loader.
2. Remove invalid placeholders or place the real installation directory earlier on PATH, within the search limits. A successful shell git --version alone does not prove that vouchfx resolves the same binary: shells can use aliases, functions and shims that this selection refuses.
3. Retry selection from an existing repository. A real git that starts and rejects the repository or reference keeps its own failure; vouchfx never retries that result with another binary.

All three selection refusals retain exit 2. A timeout or failed capture also refuses selection without using partial output; caller cancellation follows the cancellation path instead.

---

See also

• Recipes — Task-oriented examples for common scenarios.
• vouchfx-samples — Real-world sample applications and test suites demonstrating patterns.
• Common Patterns — Authoring patterns and step structure.
• Language Reference — Complete field reference for every step type.
• Technical Architecture Blueprint — How the system works (Aspire, Roslyn, memory model, verdict taxonomy, secrets).
• Project README — Building and running vouchfx, CLI reference, exit codes.