Selective Test Execution

Overview

Selective test execution is a GAIA-owned CI pipeline feature. It replaces GitHub's native paths: triggers (which cannot express transitive dependencies between stacks) with a six-stage pipeline that maps changed files to affected stacks, walks the dependency graph, applies safety escalations, and emits a GitHub Actions matrix so only the relevant jobs run.

When test_policy is absent from your config, every stack's tests run on every trigger -- identical to the behavior before selective execution existed. Adding the section is opt-in and backward-compatible.

How it works

Six scripts run in sequence during each CI invocation. Each stage can escalate the affected set to ["*"] (all stacks), but none can subtract from it. The selective-test-driver.sh script chains stages 1 through 5 into a single invocation that emits the final GitHub Actions matrix JSON on stdout. The selective-tests.yml workflow calls the driver in a plan job and fans out per-stack test jobs via the matrix.

Scenario 1: Single-stack project

Setup

A Node.js API with one stack. All source lives under src/.

# .gaia/config/project-config.yaml
stacks:
  - name: api
    path: src
    language: typescript
    test_command: npm test

What happens

Every code change touches the single stack, so detect-affected.sh always returns ["api"]. There is no dependency graph to walk and no cross-stack narrowing to apply. The CI matrix contains one job.

Before / after comparison

Takeaway

For a single-stack project the main benefit is skipping CI entirely on documentation-only changes. If your CI takes 8 minutes, that is 8 minutes saved on every README or config edit.

Scenario 2: Full monorepo

Setup

A monorepo with three independent stacks: a React frontend, a Python API, and a Go worker service. No stack imports code from another.

# .gaia/config/project-config.yaml
stacks:
  - name: frontend
    path: packages/web
    language: typescript
    test_command: npm test
  - name: backend
    path: packages/api
    language: python
    test_command: pytest
  - name: worker
    path: packages/worker
    language: go
    test_command: go test ./...

What happens

detect-affected.sh maps each changed file to its owning stack. Because no stack declares cross_refs, the dependency walk adds nothing.

Before / after comparison

Expected CI matrix output

For a change to packages/web/src/App.tsx only:

{"include":[{"stack":"frontend"}]}

Takeaway

A three-stack monorepo with no cross-dependencies sees a 66% reduction in CI jobs on single-stack changes. The savings scale linearly with the number of independent stacks.

Scenario 3: Shared library / cross-stack dependency

Setup

Same three stacks as Scenario 2, but now the frontend and backend both import from a shared utility library. The library is a fourth stack.

# .gaia/config/project-config.yaml
stacks:
  - name: shared-lib
    path: packages/shared
    language: typescript
    test_command: npm test

  - name: frontend
    path: packages/web
    language: typescript
    test_command: npm test
    cross_refs:
      - shared-lib

  - name: backend
    path: packages/api
    language: python
    test_command: pytest
    cross_refs:
      - shared-lib

  - name: worker
    path: packages/worker
    language: go
    test_command: go test ./...

What happens

When a file in packages/shared/ changes, detect-affected.sh returns ["shared-lib"]. Then cross-refs-walk.sh reads the dependency graph: frontend declares shared-lib in its cross_refs, so frontend is pulled in. backend does the same. worker has no dependency on shared-lib, so it stays out. The final affected set is ["shared-lib", "frontend", "backend"].

Before / after comparison

Expected CI matrix output

For a change to packages/shared/utils.ts:

{"include":[{"stack":"shared-lib"},{"stack":"frontend"},{"stack":"backend"}]}

What if the dependency graph is wrong?

Suppose the worker stack starts importing from shared-lib but nobody updates cross_refs. reconcile-stale-graph.sh detects the undeclared import edge and escalates the entire run to ["*"] (all stacks). The CI output will include a warning naming the undeclared edge so you can fix the config. See stale-graph escalation below.

Scenario 4: Per-trigger test policy configuration

Setup

The same four-stack project, now with a test_policy section that scopes test execution differently depending on the CI trigger.

# .gaia/config/project-config.yaml
test_policy:
  always_run:
    - shared-lib

  flaky:
    - frontend-e2e-checkout

  retry_limit: 3

  triggers:
    pr:
      exclude_tags:
        - slow
    push:
      include_stacks:
        - backend
        - worker
    schedule:
      include_tags:
        - slow
        - e2e

What each trigger does

Before / after comparison (on a PR that touches the frontend)

Important: include and exclude are mutually exclusive

For each trigger, you can specify include_stacks or exclude_stacks, but not both. The same applies to include_tags and exclude_tags. The config schema rejects combinations that specify both, with a validation error pointing to the exact location in your YAML.

Safety rails

Selective execution trades completeness for speed. The pipeline includes four safety mechanisms that prevent this trade-off from causing under-testing. Each one is explained below with a concrete example.

1. The always_run set

The test_policy.always_run array lists stacks (or suite identifiers) whose tests run on every CI invocation, regardless of which files changed. This is additive -- it can never subtract from the affected set.

test_policy:
  always_run:
    - shared-lib
    - critical-auth

Example: A developer edits only packages/web/src/Button.tsx. The affected set is ["frontend"]. After applying always_run, the set becomes ["frontend", "shared-lib", "critical-auth"]. The shared library and authentication tests run even though their code did not change.

2. Stale-graph escalation

reconcile-stale-graph.sh reuses the brownfield cross-stack import-edge detector to find undeclared dependencies. If your code imports a module from another stack but that stack is not listed in cross_refs, the dependency graph is stale. The script escalates the entire run to ["*"] (all stacks) and logs a warning naming the undeclared edge.

# Example warning on stderr:
# reconcile-stale-graph: undeclared import edge: worker -> shared-lib
# Escalating to full suite ["*"]

Why this matters: Without this check, a change to shared-lib would not trigger the worker's tests even though the worker depends on it. The escalation ensures you are never under-tested because of a stale config. Fix the config by adding the missing cross_refs entry, and subsequent runs will narrow correctly again.

3. Promotion-push wildcard

When a change is promoted to the final tier of your promotion chain, detect-affected.sh receives a --event promotion-push flag and immediately returns ["*"] without inspecting any files. Every stack's tests run on that promotion.

The full-suite tier is derived from ci_cd.promotion_chain — specifically the last tier's branch — never hard-coded. The reference selective-tests.yml reads the chain, determines the PR/push base branch, and passes --event promotion-push only when the base is that final tier. So a project that promotes feature → staging → main runs the narrowed affected-set on feature → staging (the fast dev loop) and the full suite only on staging → main (the promotion gate). A project whose final tier is named something other than main still escalates correctly. If the chain cannot be read, the workflow falls back to treating main as the final tier.

Why this matters: Promotion merges to the final tier are the last gate before production. Running the full suite here catches integration issues between stacks that were tested independently during the PR phase. The wildcard is applied before any policy narrowing, so per-trigger include_stacks filters cannot reduce a promotion push.

4. The --force-full-run escape hatch

Pass --force-full-run to apply-test-policy.sh to override all narrowing and run every stack's tests. This escalates the affected set to ["*"].

# In your CI workflow step:
apply-test-policy.sh --force-full-run --trigger pr \
  --config .gaia/config/project-config.yaml

When to use it: After a major dependency upgrade, a framework version bump, or any change where you are uncertain which stacks may be affected. The flag is a manual override, not an automatic behavior -- use it when you want explicit full coverage.

Configuring test_policy

The test_policy section lives in .gaia/config/project-config.yaml. When the section is absent, all tests run for all stacks on every trigger. Adding it opts you into selective execution.

Minimal configuration

The simplest useful configuration sets an always_run list and nothing else. All other fields have safe defaults.

# .gaia/config/project-config.yaml
test_policy:
  always_run:
    - core

Full configuration

A complete test_policy section with all fields:

# .gaia/config/project-config.yaml
test_policy:
  always_run:
    - shared-lib
    - auth-service

  flaky:
    - checkout-e2e
    - payment-integration

  retry_limit: 2

  triggers:
    pr:
      exclude_tags:
        - slow
        - nightly
    push:
      exclude_stacks:
        - docs
    schedule:
      include_tags:
        - slow
        - e2e
        - security

Field reference

Scope rule fields (per trigger)

Validating your configuration

After editing test_policy, validate the config:

/gaia-config-validate

Invalid stack references in always_run, include_stacks, or exclude_stacks produce a validation error with the exact JSONPath location. Fix the reference and re-validate.

Flaky test retries

run-with-retry.sh handles tests flagged in the test_policy.flaky list. When a flaky-tagged test fails, it is retried up to retry_limit times. If it passes on any retry, the failure is suppressed. If it exhausts all retries, the failure escalates to a real CI-blocking failure -- flaky tests never silently pass.

test_policy:
  flaky:
    - checkout-e2e
  retry_limit: 3

In this example, the checkout-e2e test gets up to 3 retries on failure. If it fails 4 times in a row (1 initial + 3 retries), the CI run fails.

What to read next

• Test Strategy Configuration -- how to configure test tiers, the test pyramid, and the Test Execution Bridge.
• Configuring CI Pipelines -- promotion chains, presets, triggers, and the layered CI model.
• CI Scenarios by Team Size -- concrete configurations for solo through enterprise setups.
• Project Shapes Overview -- how project structure affects CI and test configuration.