Sample decision review

A warning a backtest couldn’t have seen.

A concrete example of our research format: a historical financial signal, its original release, and what changed after revisions.

Financial conditionsPublic-data replayJuly 31, 2024Not client work

OVERWATCH VELDT / RESEARCH SAMPLE 01POINT-IN-TIME DECISION INTEGRITY

What did the indicator actually say that morning?

A later chart can contain revisions that did not exist when the original decision was made. We compared the historical data as available on July 31, 2024 with the revised series later in the database.

Information available then

Original Chicago Fed NFCI release

−0.58299

Our replay records the reading available on July 31 for the preceding week. It was still signalling relatively loose financial conditions.

History viewed later

Revised NFCI history

≈0.22 points

Our replay found a difference of roughly 0.22 index points, including a reversal in the week-over-week direction.

The finding

Same observation. Different information state. A model tested today could appear to have received a warning that was not available to the decision-maker then.

The decision question

Would a past risk review pass the same test using only information available then?

Our replay looked beyond one indicator. Credit spreads were beginning to tighten while market breadth was still strong. The picture on that date was mixed, not a clean automatic sell signal.

What we checked

The originally published vintage of the Chicago Fed National Financial Conditions Index (NFCI).
How the same observation appears in revised historical data.
Other evidence available at the time, including high-yield credit spreads and market-price context.
Whether the changed information state could distort an evaluation of past decisions.

Our recommendation in the sample

Before judging a historical decision, preserve the original release, its timestamp and the evidence available at the time. Mark revised observations separately. If a change can alter the reading, retest the decision using genuine point-in-time records.

What this does not prove

This was an independent public-data replay. It did not test a client's confidential trading rules. It does not prove any investment would have earned more money or that a particular trade was appropriate. The figures reflect our research reconstruction and should be independently reproduced for consequential use.

Public references

The original and revised vintage histories are publicly accessible. They allow readers to examine the kind of issue described here.

ALFRED / NFCI vintages ↗FRED / NFCI ↗FRED / High-yield spreads ↗

Source data can be revised. Exact sample numbers here come from an Overwatch research replay of public records; they are not independently audited figures.

Your decision. A clear next step.

Have something you need checked?

Send us the decision and what worries you. We'll tell you if a focused review makes sense.

Request a review ↗