/gaia-test-manual

What it does

/gaia-test-manual dispatches the manual-tester agent (Reese) to exercise a target the way a human tester would. Reese runs commands, observes output, compares expected vs. actual behavior, and produces a structured run record with per-step evidence. The skill persists the evidence artifacts (a run-record file and an exit-code log) and enforces a proof-of-execution gate before accepting a PASSED verdict.

This is agent-driven human-style verification -- complementary to, not a replacement for, the automated test suite executed by /gaia-test-run. Use this skill for exploratory testing, smoke testing, UX verification, and validating behaviors that resist automation.

When to use it

• You need a human-style walkthrough of a skill, script, workflow step, or user-facing feature with observed-vs-expected evidence.
• The behavior under test is difficult or impossible to cover with automated tests (visual polish, subjective UX, edge-case flows the automated suite does not yet cover).
• You want to produce a run record that proves a feature was exercised end-to-end before closing a story.
• You are running the sprint-review ceremony and want the execution track to include a manual-test pass for configured surfaces.

Prerequisites

• A target must be provided. Pass the name of the skill, script, feature, or workflow step to test (e.g., /gaia-test-manual gaia-changelog). The skill refuses without a target.
• At least one surface must be configured in .gaia/config/project-config.yaml for the skill to run (see Verification surfaces below). Surfaces that are not configured are automatically SKIPPED.
• For visual regression (browser surface): ImageMagick (compare) or pixelmatch must be on PATH. When neither is available, the visual check degrades to UNVERIFIED with a diagnostic.

How to invoke

/gaia-test-manual gaia-changelog
/gaia-test-manual gaia-sprint-plan
MANUAL_TEST_SURFACE=browser /gaia-test-manual login-flow

The MANUAL_TEST_SURFACE environment variable selects a specific surface. When omitted, the default surface is api.

Verification surfaces

The skill supports four verification surfaces. Each surface is keyed to a signal in .gaia/config/project-config.yaml. When the signal is absent, the surface is SKIPPED (dormant) -- it is simply not relevant to the project. SKIPPED is distinct from UNVERIFIED: SKIPPED means dormant, UNVERIFIED means the surface was exercised but evidence was insufficient.

Visual regression baseline lifecycle

When the browser surface runs, the skill captures per-breakpoint screenshots and compares them against per-story design baselines via the pixel-diff comparator.

Threshold and masking

The visual_diff section in project-config.yaml controls comparison behavior:

• threshold_percent (default 0.1) -- the maximum percentage of pixels that may differ. At-threshold passes; only strictly above fails.
• mask_regions -- rectangular regions (x, y, w, h, label) excluded from comparison before diffing. Use for dynamic content like timestamps, ads, or live data feeds.

Baseline states

• No baseline exists: the visual check records UNVERIFIED (non-blocking). A missing baseline never blocks the story-to-done transition.
• Baseline exists, diff within threshold: the visual check records PASSED.
• Baseline exists, diff exceeds threshold: the visual check records FAILED.

Approving a new baseline

After an intentional design change, update the baseline by running approve-baseline.sh --story <slug> --breakpoint <width>. The approval script requires an interactive terminal and explicit confirmation per breakpoint -- baselines are never auto-accepted. Old baselines are archived under a previous/ subdirectory and each approval is logged to baseline-approvals.log.

What it does step by step

1. Resolve target and evidence path Resolves the target argument to a concrete testable entity and derives a slug for the evidence directory.
2. Resolve surface profile Determines which surface to exercise. If the surface is not configured, the skill skips directly to the verdict step with SKIPPED status.
3. Dispatch the manual-tester agent Dispatches Reese via a fork context with read-only tools. The agent runs commands, observes output, and returns a structured run-record payload.
4. Persist evidence artifacts Writes the run-record file and exit-code log to the canonical evidence directory.
5. Enforce proof-of-execution gate Validates that both evidence files exist and are non-empty. If the verdict is PASSED but evidence is missing or empty, the verdict is downgraded to UNVERIFIED.
6. Surface verdict and findings Reports the final verdict. WARNING and CRITICAL findings are surfaced to the user; INFO findings remain in the evidence files only.
7. Record verdict on review-gate ledger Records the manual-test verdict on the extended review-gate ledger and appends to the persistent verdicts file for flakiness tracking.

Inputs

Outputs

Advisory review gate

The manual-test verdict is recorded as an advisory entry on the extended review-gate ledger. This is the 7th review gate -- advisory, not blocking. Stories with manual_verification: true in frontmatter will see a WARNING on the review-to-done transition if the latest manual-test verdict is FAILED, but the transition is not blocked. This matches the advisory-gate pattern used by other extended ledger entries.

Example session

> /gaia-test-manual gaia-changelog

Surface: api (default)
Target: gaia-changelog

Dispatching manual-tester agent (Reese)...

Run record (3 steps):
  Step 1: invoke /gaia-changelog           PASSED  (exit 0, changelog entry written)
  Step 2: verify changelog format           PASSED  (Keep a Changelog sections present)
  Step 3: verify story key cross-reference  PASSED  (story keys resolved from sprint)

Evidence written:
  run-record.md   -> .gaia/artifacts/test-artifacts/manual-test/gaia-changelog/run-record.md
  exit-code.log   -> .gaia/artifacts/test-artifacts/manual-test/gaia-changelog/exit-code.log

Proof-of-execution gate: PASSED (both evidence files present and non-empty)

Verdict: PASSED

What to run next

• If the verdict is PASSED, proceed with /gaia-review-all to complete the review cycle.
• If the verdict is FAILED, investigate the run record, fix the issue, and re-run.
• If the verdict is UNVERIFIED (missing evidence or missing baseline), supply the missing prerequisite and re-run.

Troubleshooting

"usage: /gaia-test-manual <target>"

No target was provided. Pass the name of the skill, script, or feature to test as a positional argument.

Surface reports SKIPPED

The surface is not configured in .gaia/config/project-config.yaml. This is expected for surfaces that are not relevant to your project. Configure the platform or desktop commands to enable the surface.

Visual check reports UNVERIFIED (no baseline)

No design baseline exists for this story. Capture an initial baseline by running the browser surface once, then approve it with approve-baseline.sh --story <slug> --breakpoint <width>.

Visual check reports UNVERIFIED (missing tool)

Neither ImageMagick (compare) nor pixelmatch is on PATH. Install one of them and re-run.

Verdict downgraded to UNVERIFIED

The proof-of-execution gate found that the run-record or exit-code log is missing or empty despite a PASSED agent verdict. This usually means the agent completed without producing output. Re-run and inspect the agent transcript.

Related commands