flowchart LR
C[Contract and fixture] --> R[Meaningful behavioral red]
R --> G[Minimum implementation and green]
G --> E[Regression and evidence]
Status: foundation implemented; WSR inspection/execution/security cases remain planned.
The implemented slice includes validated case catalogs, owned Python pass/failure references, committed FastAPI/uv workflow and license snapshots, explicit pinned bare-source preparation, and a Cargo integration test for the offline foundation. See the corpus guide for actual commands and native reference evidence for observed runs. Later phases below remain proposals, not implemented capabilities.
This plan defines how tests will guide WSR development. It does not approve unresolved product architecture or change the design status in PLAN.md. ROADMAP.md owns delivery gates; CHECKLIST.md records implemented behavior.
Use ordinary Cargo unit tests in source files and integration tests in each owning
crate’s tests/ directory. Testing WSR gives the implemented
commands and layout. There is no mandatory case-ID or catalog-registration step.
The INS/EXE inventory below is supporting planning material, not a prescribed task
queue or a prerequisite for starting TDD.
The dogfood inputs are the actual manual-only .github/workflows/example-*.yml files.
As the GHA translator is implemented, use these same files in Rust tests asserting the
real translation to IR without executing commands. Once execution is implemented,
use them in WSR CLI/engine tests and compare meaningful outcomes against manual native
GitHub runs. Do not substitute a YAML parse test or a simulation for either product path.
Unresolved future runtime decisions do not block independent unit/translation work;
settle the narrow interface needed for each behavior before writing its acceptance test.
Success means a developer can change WSR, run the relevant validation, and identify which documented workflow behaviors still work, which regress, and which have never been implemented.
Assumptions for this proposal:
| Layer | Inputs | Assertions | When required |
|---|---|---|---|
| Unit and CLI contract | Small values and temporary repositories | Parsing, diagnostics, exit codes, selection, isolated module behavior | Every PR |
| Workflow inspection | Committed YAML, scripts, event fixtures, expected semantic results | Discovery, source locations, dependencies, compatibility, expressions and matrix planning as implemented; no workload execution | Every PR after activation |
| Owned execution examples | Tiny committed projects copied to disposable directories | Actual outputs, failure attribution, job/step states and artifacts | Every relevant PR on a qualified backend after activation |
| Security and lifecycle | Bounded adversarial fixtures and synthetic credentials | Grants/denials, fail-closed behavior, cleanup, host/control-state protection | Every relevant backend/policy PR; full suite before release |
| External compatibility | Exact upstream commits and explicit invocation context | Selected real workflow behavior and declared unsupported requirements | Manual first; scheduled after stabilization; selected release gate |
| GitHub reference | Owned manually dispatched example workflows | Compare selected semantic outcomes against GitHub’s execution | Manual and before promoting a compatibility claim |
An intentionally failing workload can produce a passing WSR test: the assertion is that WSR reports its failure correctly. Conversely, a workload passing does not prove that WSR enforced isolation.
.github/workflows/
example-shell.yml # Manual shell pass/failure reference
example-python.yml # Manual Python pass/failure reference
example-job-outputs.yml # Future example, added with that feature
validate-wsr.yml # Future host-driven suite orchestration
examples/
README.md
python-basic/
README.md
pyproject.toml
tests/test_example.py
job-outputs/ # Added when needed
service-postgres/ # Deferred until services are supported
tests/ # Root test data
README.md
cases.toml # Small catalog of named cases
fixtures/
inspection/
security/
events/
expected/ # Reviewed semantic results/diagnostics
ecosystem/
cases.toml # Upstream revisions, selections, provenance
crates/wsr-gha/tests/
workflow_examples.rs # Manual-trigger guard/native shell reference
translation.rs # Future real GHA-to-IR tests using example YAML
crates/wsr/tests/
cli.rs # Existing executable contract
corpus.rs # Implemented foundation/native-reference bridge
inspection.rs # Future CLI inspection integration tests
examples.rs # Future owned execution tests
ecosystem.rs # Explicit opt-in external cases
support/mod.rs # Shared helpers only when actually shared
Root tests/ contains data and documentation. Cargo discovers integration test drivers in a
package’s tests/, such as crates/wsr/tests/; root data is not automatically executed.
Resolve test data from CARGO_MANIFEST_DIR, not the caller’s working directory.
Use a single root data directory; provider/backend unit tests can consume the same fixtures.
Put fetched sources, isolated workspaces and local results under ignored target/wsr-tests/.
Use immutable cached source snapshots plus a separate fresh writable copy per execution.
A developer may manually supply a local repository snapshot later; required cases always use the
recorded commit and must not mutate that developer’s checkout.
The existing small catalog supports native references and preserved inputs; it does not replace Rust tests or require every new behavior/workflow to be registered. The implemented loader is Python 3.11+ using standard-library TOML parsing, called by xtask and a Rust integration test. The current schema covers IDs, status, kind, input, expectations and native mode; pinned source metadata lives in the ecosystem catalog. Add the remaining fields below when their executable lanes need them, rather than inventing unresolved runtime configuration now. Each eventual case records:
planned, active, or required, with the feature gate that promotes the case.
active identifies the current test-first implementation slice; it is not passing coverage.Validate unknown fields, duplicate IDs, missing inputs and invalid expectations. A catalog entry is inventory, not automatically a passing test. Each required case must have executable assertions; report the IDs exercised and fail if a selected required case has no implementation or disappears.
Results distinguish passed, failed, planned/not run and infrastructure failure. Required cases cannot succeed by skipping, accepting any nonzero exit code, or matching the current generic “not implemented yet” message. Missing prerequisites in a selected required lane fail that lane. The ordinary suite may exclude opt-in lanes, but must report that those lanes were not exercised. Do not use ignored tests as evidence of coverage.
An active case must be selected explicitly and its real failure reported during development. Its expected product failure never becomes a passing assertion or an allowed-failure CI gate. Existing required cases remain enforced while the active slice is developed. Promote the active case to required, and include its lane in the applicable gate, before calling the feature complete. This permits a green supported baseline without pretending the planned backlog already works.
Assert semantics rather than entire unstable log streams. Normalize only incidental fields such as temporary paths and timestamps; retain meaningful source locations, grants and state changes. Expected results require review and must not be rewritten automatically when a test fails.
Add these incrementally with their corresponding phase, rather than creating empty directories or a large collection of disabled test bodies upfront.
| IDs | Case | Initial validation and later promotion |
|---|---|---|
| INS-001 | Valid manual-dispatch workflow | Discover job/steps and preserve source traceability |
| INS-002 | Invalid YAML and invalid workflow structure | Specific parse/validation diagnostics with source locations |
| INS-003 | Unknown action/platform/required feature | Explicit unsupported/unknown classification; block unsupported execution |
| INS-004 | Dependency cycle or missing dependency | Reject invalid plan |
| INS-005 | Inspection side-effect sentinel | Repository scripts do not execute; source and host sentinel remain unchanged |
| INS-006 | Two-value matrix and event/input conditions | Correct expansion/binding as those semantics become available |
| EXE-001 | Tiny Python project | One passing test; selected Python and dependency identities reported |
| EXE-002 | Same project with controlled assertion failure | Failed test maps to failed step/job and failing command exit |
| EXE-003 | Job output consumed by dependent job | Correct output propagation, ordering and failure/skip behavior |
| EXE-004 | Artifact produced and retrieved | Exact expected contents and bounded destination paths |
| SEC-001 | Minimal component with explicit grants | Permitted operation works; absent grant is denied |
| SEC-002 | System filesystem/network access | In-grant access works; outside access is blocked |
| SEC-003 | Policy and synthetic credential probes | Repository input cannot grant itself access or read unprovided host credentials |
| SEC-004 | Traversal/symlink/archive and shared-state probes | No escape or control-state modification across relevant host boundaries |
| LIFE-001 | Timeout/cancellation with a child process | Run ends within budget; descendants and temporary resources are cleaned up |
| BACKEND-001 | Required enforcement unavailable | Reject before workload side effects; no permissive fallback |
| EXE-005 | Tiny app plus PostgreSQL service | Connectivity, readiness, job scope and teardown; later milestone |
Security cases use bounded disposable resources and synthetic secrets. The test supervisor must observe denials and cleanup from outside the workload boundary; do not trust workload-reported success as proof of enforcement. Use positive controls so a missing test setup cannot look like a successful denial. Never execute escape probes uncontained on a personal host.
Each owned project explains what it teaches, prerequisites, expected success/failure, current WSR support, and how to invoke it once an actual product command exists. Do not invent stable CLI flags or TOML configuration before those contracts are decided.
Reference workflows in .github/workflows/example-*.yml use workflow_dispatch only initially.
They execute the project’s commands directly on GitHub. Local WSR tests consume those same workflow
files and projects. There is one canonical workflow per example; avoid maintained duplicate YAML.
Nested projects may contain only the additional files genuinely needed for repository discovery.
Use typed manual inputs for deterministic success/failure variations. Give intentionally failing
reference runs clear names and document their expected failed conclusion; do not hide failures with
continue-on-error. Keep examples free of publishing, deployment and real-secret requirements.
Pin action commits and relevant dependencies, and record image/runtime drift that pins cannot remove.
workflow_dispatch permits native GitHub execution; it does not make a workflow local-only.
Repository discovery rules must allow the WSR harness to select an example explicitly without
accidentally executing normal release workflows.
The orchestration workflow builds WSR and invokes the host-side test harness. Example payloads must not invoke WSR’s full test suite or recursively dispatch validation. Once engine reuse inside Actions is supported, add a separate path that runs examples through WSR there. Keep the native GitHub reference as an independent comparison.
Compare selected outcomes, not byte-identical logs: success/failure, skip propagation, matrix membership, outputs and artifact contents. Security policy is WSR-specific and needs its own oracle. A stored GitHub reference records source revision, inputs and run URL; it is not timeless proof.
Start with FastAPI for external inspection and uv for a later inspection/compatibility challenge. The exact commits and runnable selections must be inspected and pinned during onboarding; do not assume the default branch or entire workflow can execute. Defer Chromium until a specific relevant workflow and affordable objective are identified. No external support claim is made by this plan.
For each upstream case:
A network/setup failure is infrastructure failure, not WSR incompatibility or a successful test. When WSR fully supports a selected case, a new failure is a regression until diagnosed.
Deliver tests/README.md, the minimal catalog schema/loader, INS-001 through INS-005 behavioral
specifications, the first active inspection fixture, the Python example, and its manual native
GitHub workflow. Create remaining executable fixtures as their slices become active.
Keep unimplemented product cases marked planned; do not add fake green execution tests.
Use existing Rust dependencies/test helpers where sufficient; no new crate is required.
Use the section 11 handoff template for concrete behaviors; catalog IDs are optional.
Gate: catalog validation rejects malformed/duplicate/missing entries; existing tests remain intact; the Python example’s pass/fail variants work directly in a declared environment. Before treating GitHub as a reference, execute both variants manually and record their actual run evidence. A workflow file alone is not reference evidence.
Drive GHA-to-IR translation through ordinary Rust tests on the canonical manual workflow files. Assert the supported jobs/steps/trigger mapping and IR semantics without executing commands. Add inspection CLI tests when that path is delivered; the INS inventory suggests boundary cases, but is not a required sequence. Test discovery, provider failures, diagnostics and source mappings; use deterministic local provider artifacts for normal tests rather than implicit downloads. Provider bootstrap tests separately cover signature/digest rejection, pins, atomic installation, offline cache hits and missing pinned artifacts using controlled sources.
Gate: required inspection cases pass offline without executing repository commands. Product stubs fail these assertions. Provider/backend implementation decisions remain owned by PLAN.md.
After the corresponding runtime/backend decisions, activate SEC-001 for components, then SEC-002 through SEC-004, LIFE-001 and BACKEND-001 for system execution. Qualify each host/backend separately. Add a trusted outer test supervisor and bounded execution/cleanup checks before adversarial runs.
Gate: positive controls work, unauthorized probes fail, cancellation cleans up, and missing enforcement blocks execution. A backend without this evidence is not a supported secure backend.
Activate EXE-001 and EXE-002 through the complete provider/planner/executor path. This includes support for every selected setup action, or a clearly labeled smaller shell-only workflow until those actions are supported. Do not replace unsupported actions silently in the canonical example. Implement the required matrix/input semantics and activate INS-006 as appropriate.
Gate: native GitHub reference and WSR agree on selected workload outcomes; WSR identifies the failing step and test output, returns the correct exit status, and reports environment/grants. Run on Linux and on macOS-hosted Linux once those backends are qualified.
Activate EXE-003 and EXE-004 with job/expression/artifact support; add EXE-005 only with services. For each feature add both normal behavior and a meaningful failure case, then require the case in its relevant validation lane. Maintain WSR’s ordinary CI as an independent test runner.
Gate: documented dependency, failure, skip, output and cleanup semantics hold across the supported backends. Update CHECKLIST.md only with links to working tests and observed evidence.
Onboard pinned FastAPI inspection, then selected execution when supported. Add uv inspection as another independent compiler input; choose an executable uv case only after inspecting its needs. Introduce explicit preparation and execution commands only when the first external case needs them; keep them in existing xtask rather than creating another development CLI.
Gate: clean preparation reproduces the pinned case; pass/fail evidence is retained; required external cases run in a scheduled lane and before releases that claim their compatibility. Do not mark a whole repository supported because one job passes.
Existing cargo xtask test and cargo xtask ci retain their roles. Ordinary Cargo tests include
fast deterministic unit/CLI/inspection cases as activated. Heavy/backend/external tests require
explicit selection and preparation; ordinary tests do not install tools or fetch external repos.
The foundation now provides cargo xtask corpus, --list, --case ID and --prepare REPOSITORY.
These commands validate native references/snapshot integrity or prepare sources; they do not run
WSR workflows. Exact execution-lane syntax remains deferred.
Use explicit lane selection as well as affected-path rules: shared engine/policy changes must not escape validation just because example files were unchanged. Preserve current CI and add lanes incrementally. Scheduled execution is configured during implementation, not by this proposal.
Persist case ID, WSR revision, input digest/upstream SHA, host/job platform, backend/runtime/provider identities, effective grants, duration and outcome. Retain relevant logs/artifacts on failures with synthetic secrets redacted. A small versioned JSON result format is sufficient; no dashboard needed.
A feature is complete only when its behavior tests run, pass and are enforced by the appropriate CI gate. Standard Cargo discovery is sufficient; catalog registration is optional. Planned fixtures, ignored tests, documentation and source compilation are not completion. Unsupported behavior must receive a precise diagnostic and must never become an implicit success.
The corpus foundation is supporting tooling. Product progress is measured by real Rust behavior tests, then translation and execution of the canonical manual workflows. The original first execution demonstration was the tiny Python project’s controlled failure, diagnosis and repaired success. The owner now prioritizes the bounded Java E2E worker; retain the Python references and regressions while developing that slice. Native references and actual WSR execution remain distinct. Pinned external cases challenge assumptions that owned examples might miss.
Each task is a small observable behavior with a descriptive Rust test name, not an entire subsystem. A catalog case ID is optional; a normal unit or integration test is sufficient. A maintainer or planning agent prepares the following handoff before assigning implementation:
Behavior and test name (optional catalog ID):
Accepted decision/specification links:
Public interface and expected outcome:
Input fixture/event and relevant negative or boundary case:
Existing test/validation paths, or the first test to add:
Exact focused command and required environment/preparation:
Expected red failure and why it demonstrates the missing behavior:
Existing required regression commands:
Permitted scope and explicit non-goals:
Unresolved prerequisites, or "none":
Completion evidence and CI lane:
Fill commands from actual implemented tooling; a placeholder command is not an executable handoff. A ready slice needs enough settled semantics to distinguish correct behavior from an arbitrary implementation. If interface, policy or backend decisions are unresolved, report the specific decision needed rather than inventing it to satisfy a test. Existing architecture gates remain in force; tests cannot authorize an otherwise undecided design.
Tests and validation should be reusable task inputs. A test may be authored in a preparatory change and handed to another agent with its observed red result. A pending test may remain on a work branch, but must not break required main-branch CI. Either land the vertical slice together, or keep the active case in an explicitly selected development lane until implementation lands. Do not introduce ignored tests or expected-failure wrappers to simulate this distinction.
If a supplied test already passes, verify that it reaches the intended path and covers the acceptance behavior. Report that result; do not manufacture a red run or add unnecessary product code. Identify an uncovered requirement only if the task calls for one. Distinguish new behavior driven by TDD from already-working behavior confirmed by validation.
Agents must not delete/ignore required cases, relax assertions, broaden normalization, refresh expected snapshots, downgrade case status or weaken CI selection merely to make a change green. When an expectation is demonstrably wrong, report the conflicting specification and proposed correction for review. An intentional behavior change needs an explicit specification update and reviewed expectation changes; do not treat generated current output as the oracle.
Behavioral tests should survive internal refactoring. Test outcomes, state transitions and public diagnostics rather than private method names, internal object layouts or call counts. Add focused unit tests for complex rules where helpful, while retaining end-to-end assertions for product claims. Add a second input or a negative case when needed to distinguish real behavior from a hard-coded answer; do not require redundant tests merely to increase test counts.
When an external workflow exposes a bug, reproduce it in a small owned regression fixture first where possible, observe red, fix it, then rerun both the fixture and the external case. Preserve the original external case as evidence that the reduction still represents a real integration.
The implementing agent reports case IDs, specification links, the focused red command/result, the green command/result, regression results, changed scope and any unexercised lanes. Include WSR/input revisions and backend identities where relevant. Keep this evidence in the task/PR record; durable compatibility evidence follows section 9. Do not commit transient logs as source.
The handoff and completion record must let another agent reproduce validation without chat history. Passing the supplied tests is necessary; fulfilling the stated behavior and preserving the existing contract are also required. Tests cannot establish untested compatibility or security.
Planning/provider tests assert semantic lowering and diagnostics without workload side effects. Provisioning/cache tests assert exact identity, closure, approved misses, cached use without upstream calls, revocation/corruption and unsupported-tool admission. Engine tests assert ready/blocked states, global count/capacity admission and cancellation/recovery across repositories. Backend tests prove allowed and denied capabilities. Observability tests cover scoped resource attribution and queue reasons; evidence/connector tests cover claim trust, retained results and duplicate delivery.
These are planned lanes, not observed coverage. Native references, compilation, WSR execution, isolation, reproducibility and publication are separate claims. D-025/D-027 dynamic-discovery limits must be tested rather than bypassed by guessed dependencies or permissive replay.