Ln 636 Manual Test Auditor

> **Paths:** File paths (`references/`, `../ln-*`) are relative to this skill directory.

# Manual Evidence Auditor (L3 Worker)

**Type:** L3 Worker

Specialized worker auditing whether manual test scripts produce reproducible, useful evidence.

## Purpose & Scope

- Audit **Manual Evidence** (Category 7: Medium Priority)
- Evaluate bash test scripts in `tests/manual/` against quality dimensions
- Emit `REWRITE_MANUAL_EVIDENCE` or `KEEP_MANUAL_EVIDENCE`
- Calculate compliance score (X/10)

## Inputs

**MANDATORY READ:** Load `references/audit_worker_core_contract.md`.

Receives `contextStore` with: `tech_stack`, `testFilesMetadata` (filtered to `type: "manual"`), `codebase_root`, `output_dir`.

Manual test metadata includes: `suite_dir`, `has_expected_dir`, `harness_sourced`.

## Workflow

Detection policy: use two-layer detection (candidate scan, then context verification); load `references/two_layer_detection.md` only when the verification method is ambiguous.

1) **Parse Context:** Extract manual test file list, output_dir, codebase_root from contextStore
2) **Discover Infrastructure:** Detect shared infrastructure files:
   - `tests/manual/config.sh` -- shared configuration
   - `tests/manual/test_harness.sh` -- shared test framework (if exists)
   - `tests/manual/test-all.sh` -- master runner
   - `tests/manual/TEMPLATE-*.sh` -- test templates (if exist)
   - `tests/manual/regenerate-golden.sh` -- golden file regeneration (if exists)
3) **Scan Scripts (Layer 1):** For each manual test script, check 7 quality dimensions (see Audit Rules)
3b) **Context Analysis (Layer 2 -- MANDATORY):** For each candidate finding, ask:
   - Is this a setup/utility script (e.g., `00-setup/*.sh`, `tools/*.sh`)? Setup scripts have different requirements -- skip harness/golden checks
   - Is this a master runner (`test-all.sh`)? Master runners orchestrate, not test -- skip all checks except fail-fast
   - Does the project not use a shared harness at all? If no `test_harness.sh` exists, harness adoption check is N/A
4) **Collect Findings:** Record violations with severity, location (file:line), effort, action, recommendation
5) **Calculate Score:** Count violations by severity, calculate compliance score (X/10)
6) **Write Report:** Build full markdown report in memory per `references/templates/audit_worker_report_template.md`, write to `{output_dir}/ln-636--global.md` in single Write call
7) **Return Summary:** Return minimal summary to coordinator (see Output Format)

## Audit Rules

### 1. Harness Adoption

**What:** Test script uses shared framework (`run_test`, `init_test_state`) instead of custom assertion logic

**Detection:**
- Grep for `run_test`, `init_test_state` in script
- If absent AND script contains custom test loops/assertions -> custom logic
- If `test_harness.sh` does not exist in project -> skip this check entirely

**Severity:** **HIGH** (custom logic = maintenance burden, inconsistent reporting)

**Recommendation:** Refactor to use shared `run_test` from test_harness.sh
**Action:** `REWRITE_MANUAL_EVIDENCE`

**Effort:** M

### 2. Golden File Completeness

**What:** Test suite has `expected/` directory with reference files matching test scenarios

**Detection:**
- Check if suite directory has `expected/` subdirectory
- Compare: number of test scenarios (grep `run_test` calls) vs number of expected files
- If test uses `diff` against expected files but expected dir is missing -> finding

**Layer 2:** Not all tests need golden files. Tests validating HTTP status codes, timing, or dynamic data may legitimately skip golden comparison -> skip if test has no `diff` or comparison against files

**Severity:** **HIGH** (no golden files = no regression detection for output correctness)

**Recommendation:** Add expected/ directory with reference output files
**Action:** `REWRITE_MANUAL_EVIDENCE`

**Effort:** M

### 3. Config Sourcing

**What:** Script sources shared `config.sh` for consistent configuration

**Detection:**
- Grep for `source.*config.sh` or `. .*config.sh`
- If absent -> script manages its own BASE_URL, tokens, etc.

**Layer 2:** If script is self-contained utility (e.g., `tools/*.sh`) -> skip

**Severity:** **MEDIUM**

**Recommendation:** Add `source "$THIS_DIR/../config.sh"` for shared configuration
**Action:** `REWRITE_MANUAL_EVIDENCE`

**Effort:** S

### 4. Fail-Fast Compliance

**What:** Script uses `set -e` and returns exit code 1 on failure

**Detection:**
- Grep for `set -e` (or `set -eo pipefail`)
- Check that failure paths lead to non-zero exit (not swallowed by `|| true` everywhere)

**Severity:** **HIGH** (silent failures mask broken tests)

**Recommendation:** Add `set -e` at script start, ensure test failures propagate
**Action:** `REWRITE_MANUAL_EVIDENCE`

**Effort:** S

### 5. Template Compliance

**What:** Script follows project test templates (TEMPLATE-api-endpoint.sh, TEMPLATE-document-format.sh)

**Detection:**
- If TEMPLATE files exist in `tests/manual/`, check structural alignment:
  - Header comment block with description, ACs tested, prerequisites
  - Standard variable naming (`THIS_DIR`, `EXPECTED_DIR`)
  - Standard setup pattern (`source config.sh`, `check_jq`, `setup_auth`)
- If NO templates exist in project -> skip this check entirely

**Layer 2:** Older scripts written before templates may diverge. Flag as MEDIUM, not HIGH

**Severity:** **MEDIUM**

**Recommendation:** Align script structure with project TEMPLATE files
**Action:** `REWRITE_MANUAL_EVIDENCE`

**Effort:** M

### 6. Idempotency

**What:** Script can be rerun safely without side effects from previous runs

**Detection:**
- Grep for cleanup patterns: `trap.*EXIT`, `rm -f`, `cleanup` functions
- Check for temp file creation without cleanup
- Check for hardcoded resource names that would conflict on rerun (e.g., creating user with fixed email without checking existence)

**Layer 2:** Scripts that only READ data (GET requests, queries) are inherently idempotent -> skip

**Severity:** **MEDIUM**

**Recommendation:** Add cleanup trap or use unique identifiers per run
**Action:** `REWRITE_MANUAL_EVIDENCE`

**Effort:** S-M

### 7. Documentation

**What:** Test suite directory has README.md explaining purpose and prerequisites

**Detection:**
- Check if suite directory (`NN-feature/`) contains README.md
- If missing -> finding

**Layer 2:** Setup directories (`00-setup/`) and utility directories (`tools/`) may not need README -> skip

**Severity:** **LOW**

**Recommendation:** Add README.md with test purpose, prerequisites, usage
**Action:** `REWRITE_MANUAL_EVIDENCE`

**Effort:** S

## Scoring Algorithm

**MANDATORY READ:** Load `references/audit_scoring.md`.

**Severity mapping:**
- Missing harness adoption (when harness exists), No golden files (when expected-based), No fail-fast -> HIGH
- Missing config sourcing, Template divergence, No idempotency -> MEDIUM
- Missing README -> LOW

## Output Format

**MANDATORY READ:** Load `references/templates/audit_worker_report_template.md`.

Write JSON summary per `references/audit_summary_contract.md`. In managed mode the caller passes both `runId` and `summaryArtifactPath`; in standalone mode the worker generates its own run-scoped artifact path per shared contract.

Write report to `{output_dir}/ln-636--global.md` with `category: "Manual Evidence"` and checks: harness_adoption, golden_file_completeness, config_sourcing, fail_fast_compliance, template_compliance, idempotency, documentation.

Return summary per `references/audit_summary_contract.md`.

When `summaryArtifactPath` is absent, write the standalone runtime summary under `.hex-skills/runtime-artifacts/runs/{run_id}/evaluation-worker/{worker}--{identifier}.json` and optionally echo the same summary in structured output.
```
Report written: .hex-skills/runtime-artifacts/runs/{run_id}/audit-report/ln-636--global.md
Score: X.X/10 | Issues: N (C:N H:N M:N L:N)
```

## Critical Rules

Apply the already-loaded `references/audit_worker_core_contract.md`.

- **Do not auto-fix:** Report only
- **Effort realism:** S = <1h, M = 1-4h, L = >4h
- **Skip when empty:** If no `tests/manual/` directory exists, return score 10/10 with zero findings
- **Exclude non-test files:** Skip `config.sh`, `test_harness.sh`, `test-all.sh`, `regenerate-golden.sh`, `TEMPLATE-*.sh`, files in `tools/`, `results/`, `test-runs/`
- **Context-aware:** Setup scripts (`00-setup/`) have relaxed requirements (no golden files, no harness needed)
- **Unique angle:** Audit only manual test evidence. Do not judge automated test value, E2E priority, product behavior, missing coverage, trustworthiness, oracle strength, or structure.
- **Action required:** Use `REWRITE_MANUAL_EVIDENCE` for findings. Summarize compliant scripts as `KEEP_MANUAL_EVIDENCE` in the report summary, not as findings.

## Definition of Done

Apply the already-loaded `references/audit_worker_core_contract.md`.

- [ ] contextStore parsed successfully (including output_dir)
- [ ] Manual test infrastructure discovered (config.sh, harness, templates)
- [ ] All 7 checks completed per test script
- [ ] Layer 2 context analysis applied (setup/utility exclusions)
- [ ] Findings collected with severity, location, effort, action, recommendation
- [ ] Score calculated using penalty algorithm
- [ ] Report written to `{output_dir}/ln-636--global.md` (atomic single Write call)
- [ ] Summary written per contract

## Reference Files

- **Audit output schema:** `references/audit_output_schema.md`

---
**Version:** 1.0.0
**Last Updated:** 2026-03-13