Signal Postmortem

# Signal Postmortem

## Overview

Signal Postmortem records and analyzes the outcomes of trading signals generated by the edge pipeline, screeners, and other skills. It compares predicted edge direction against 5-day and 20-day realized returns, categorizes outcomes (true positive, false positive, missed opportunity, regime mismatch), and generates feedback for edge-signal-aggregator weight adjustments and skill improvement backlog entries.

## When to Use

- After a trade has been closed and you want to record the outcome
- When reviewing a batch of signals that have reached their holding period (5 or 20 days)
- To identify systematic false positive patterns from specific skills
- To generate feedback for edge-signal-aggregator weight calibration
- When building a skill improvement backlog from decision quality metrics
- For periodic (weekly/monthly) signal quality audits

## Prerequisites

- Python 3.9+
- FMP API key (optional, for fetching realized returns if not provided manually)
- Standard library + `requests` for API calls
- Input: signal records in JSON format (from edge-signal-aggregator or screener outputs)

### API Key Setup (Optional)

If you want to automatically fetch price data for return calculations, set up the FMP API key:

```bash
export FMP_API_KEY=your_api_key_here
```

Alternatively, pass the key via command line with `--api-key YOUR_KEY`. Without an API key, you can still record outcomes manually by providing `--exit-price` and `--exit-date`.

## Workflow

### Step 1: Prepare Signal Records

Gather closed or matured signal records. Each record should include:
- `signal_id`: Unique identifier
- `ticker`: Stock symbol
- `signal_date`: Date signal was generated
- `predicted_direction`: LONG or SHORT
- `source_skill`: Which skill generated the signal
- `entry_price`: Price at signal generation (optional, for manual override)

```bash
# Example: List signals ready for postmortem (5+ days old)
python3 skills/signal-postmortem/scripts/postmortem_recorder.py \
  --list-ready \
  --signals-dir state/signals/ \
  --min-days 5
```

### Step 2: Record Outcomes

Run the postmortem recorder to fetch realized returns and classify outcomes.

```bash
python3 skills/signal-postmortem/scripts/postmortem_recorder.py \
  --signals-file state/signals/aggregated_signals_2026-03-10.json \
  --holding-periods 5,20 \
  --output-dir reports/
```

For manual outcome recording (when price data is already available):

```bash
python3 skills/signal-postmortem/scripts/postmortem_recorder.py \
  --signal-id sig_aapl_20260310_abc \
  --exit-price 178.50 \
  --exit-date 2026-03-15 \
  --outcome-notes "Closed at target, +3.2% in 5 days" \
  --output-dir reports/
```

### Step 3: Classify Outcomes

The recorder automatically classifies each signal into one of four categories:

| Category | Definition |
|----------|------------|
| TRUE_POSITIVE | Predicted direction matched realized return sign |
| FALSE_POSITIVE | Predicted direction opposite to realized return |
| MISSED_OPPORTUNITY | Signal not taken but would have been profitable |
| REGIME_MISMATCH | Signal failed due to market regime change |

Classification rules are documented in `references/outcome-classification.md`.

### Step 4: Generate Feedback Files

Generate feedback for downstream consumers:

```bash
# Generate weight adjustment suggestions for edge-signal-aggregator
python3 skills/signal-postmortem/scripts/postmortem_analyzer.py \
  --postmortems-dir reports/postmortems/ \
  --generate-weight-feedback \
  --output-dir reports/

# Generate skill improvement backlog entries
python3 skills/signal-postmortem/scripts/postmortem_analyzer.py \
  --postmortems-dir reports/postmortems/ \
  --generate-improvement-backlog \
  --output-dir reports/
```

### Step 5: Review Summary Statistics

Generate aggregate statistics by skill, by ticker, and by time period:

```bash
python3 skills/signal-postmortem/scripts/postmortem_analyzer.py \
  --postmortems-dir reports/postmortems/ \
  --summary \
  --group-by skill,month \
  --output-dir reports/
```

## Output Format

### Postmortem Record (JSON)

```json
{
  "schema_version": "1.0",
  "postmortem_id": "pm_sig_aapl_20260310_abc",
  "signal_id": "sig_aapl_20260310_abc",
  "ticker": "AAPL",
  "signal_date": "2026-03-10",
  "source_skill": "edge-signal-aggregator",
  "predicted_direction": "LONG",
  "entry_price": 172.50,
  "realized_returns": {
    "5d": 0.032,
    "20d": 0.058
  },
  "exit_price": 178.50,
  "exit_date": "2026-03-15",
  "holding_days": 5,
  "outcome_category": "TRUE_POSITIVE",
  "regime_at_signal": "RISK_ON",
  "regime_at_exit": "RISK_ON",
  "outcome_notes": "Clean breakout, held through minor pullback",
  "recorded_at": "2026-03-17T10:30:00Z"
}
```

### Weight Feedback (JSON)

```json
{
  "schema_version": "1.0",
  "generated_at": "2026-03-17T10:35:00Z",
  "analysis_period": {
    "from": "2026-02-01",
    "to": "2026-03-15"
  },
  "skill_adjustments": [
    {
      "skill": "vcp-screener",
      "current_weight": 1.0,
      "suggested_weight": 0.85,
      "reason": "15% false positive rate in RISK_OFF regime",
      "sample_size": 42
    }
  ],
  "confidence": "MEDIUM",
  "min_sample_threshold": 20
}
```

### Skill Improvement Backlog Entry (YAML)

```yaml
- skill: vcp-screener
  issue_type: false_positive_cluster
  severity: medium
  evidence:
    false_positive_rate: 0.15
    sample_size: 42
    regime_correlation: RISK_OFF
  suggested_action: "Add regime filter or reduce signal confidence in RISK_OFF"
  generated_by: signal-postmortem
  generated_at: "2026-03-17T10:35:00Z"
```

### Summary Report (Markdown)

Reports are saved to `reports/` with filenames `postmortem_summary_YYYY-MM-DD.md`.

## Resources

- `scripts/postmortem_recorder.py` -- Records individual signal outcomes
- `scripts/postmortem_analyzer.py` -- Generates feedback and summary statistics
- `references/outcome-classification.md` -- Classification rules and edge cases
- `references/feedback-integration.md` -- How to integrate feedback with downstream skills

## Key Principles

1. **Honest Attribution** -- Every outcome is attributed to its source skill for accountability
2. **Regime Awareness** -- Regime context is recorded to distinguish skill failure from market regime shifts
3. **Minimum Sample Size** -- Weight adjustments require 20+ signals for statistical validity
4. **Feedback Loop Closure** -- Results flow back to improve both signal aggregation and skill quality