calibrate-first

Calibrate First

Overview

A backtest result is only as good as the instrument that produced it. This skill makes Claude measure the instrument before it measures the signal. It drives assay-lite, a single-file, offline Python tool. floor injects synthetic signals of known strength into the user's own returns to find the smallest effect the data can detect. test charges every run to an append-only registry before computing, and a result below the floor is reported as UNDECIDABLE rather than "the signal doesn't work".

When to Use

• The user asks whether a trading signal, factor or ranking "works", is "significant", or has an "edge".
• The user wants to read an information coefficient (IC) or a cross-sectional backtest result, especially a null one.
• The user is about to try several variants of an idea and pick the best one.

Do not use it for single-asset timing signals, intraday data, live trading or position sizing. assay-lite does not cover them.

How It Works

Use only a previously reviewed local assay_lite.py at an explicit absolute path, verified against commit 78104a4356541affe2a469dfdf69cb1e418eb0c5, together with a fully locked and reviewed numpy/pandas/scipy environment. This skill does not download code or resolve dependencies. If absent, stop before execution and return the input schema and analysis plan. Do not substitute a same-named project file. A separate installation task must stage and inspect every file and dependency with installation hooks disabled, verify immutable bytes, then obtain approval before executing or writing the trial registry. uv caching is not isolation.

Four rules:

1. Run floor before interpreting any result. Until the floor is measured for these prices and this horizon, a non-significant result means nothing either way.
2. Never compare p with a conventional 0.05. The only bars are the ones the registry prints: 0.05 divided by every trial ever charged. "Bar for the last trial charged" and "the NEXT test will face" are different bars. Quote the one that matches the question.
3. Trials are never refunded. Re-running a signal, tweaking a parameter or trying another horizon is a new trial. Never suggest deleting registry lines or using a fresh --home to reset the count.
4. Below the floor is UNDECIDABLE, not "no effect". Say "too small for this data to see", never "doesn't work".

Procedure:

1. Check the inputs. Prices are date,asset,close. The signal is date,asset,signal, known at that date's close.
2. Run breadth, and state the effective number of independent bets next to the asset count.
3. Run floor for the exact prices file and horizon you will test.
4. Write down in one sentence what you expect and why, then run test once.
5. Report the verdict line verbatim, with IC, p, the bar it faced, the dispersion ratio and the scaled floor.

Examples

python /absolute/reviewed/assay_lite.py demo          # synthetic data; shows PASS, UNDECIDABLE, and a scaled floor
python /absolute/reviewed/assay_lite.py breadth  --prices prices.csv
python /absolute/reviewed/assay_lite.py floor    --prices prices.csv --horizon 1
python /absolute/reviewed/assay_lite.py test     --prices prices.csv --signal signal.csv --name momentum_12_1 --horizon 1
python /absolute/reviewed/assay_lite.py registry

Limitations

• Cross-sectional and daily only, using Spearman IC. No timing signals and no portfolio simulation.
• No transaction costs. A PASS says nothing about profit after costs.
• Bonferroni is conservative when trials are correlated.
• The floor is calibrated with independent noise and then scaled up by each signal's daily-IC dispersion. That corrects for factor structure on average, not exactly.
• Nothing here is investment advice. A PASS is evidence of a ranking effect, not permission to trade.

Security & Safety Notes

• assay-lite makes no network calls and needs no API keys. It reads the CSVs the user points it at and writes only to its own .assay-lite/ folder.
• The registry is append-only on purpose. Don't edit or delete it on the user's behalf.