The easiest backtest to produce is one that confirms what you already believe. The useful backtest is designed so that a weak idea has many honest ways to fail.
Freeze the question before exploring the answer
Write down the signal definition, eligible assets, horizon, entry timing, outcome rule, cost model, and evaluation metric before reading the test result. Every change made after seeing performance creates another opportunity to fit noise.
Keep an append-only trial record. Changing a lookback, threshold, basket, label, or filter is a new trial, not a correction to history. This matters because the best result from many attempts will look better than the underlying idea deserves.
"Buy when the line looks strong" cannot be replayed consistently. A backtest needs deterministic inputs, timing, and outcomes before it can produce evidence.
Use point-in-time data and closed decisions
Each historical decision should see only the information available at that timestamp. Common leakage paths include revised macro data, a universe chosen from survivors, a feature normalized with future values, and an entry that assumes the same close used to trigger the signal.
Record event time, observed time, and availability time when those differ. Use the same feature and decision code in live evaluation and replay wherever possible. A separate research implementation can drift into a flattering approximation without anyone noticing.
Split time instead of shuffling it
Random train-test splits break the order of markets and can place nearly identical overlapping outcomes on both sides of the boundary. Use forward-moving temporal folds. Leave a gap around each boundary when labels overlap the split, and reserve a final untouched period for the few questions important enough to deserve it.
Repeatedly checking the same holdout turns it into training data. Treat holdout access as a limited research budget. If a proposed change has already learned from the answer, it no longer has a clean final exam.
Model the friction the signal would face
Gross returns can make a short-horizon crypto signal look viable while realistic costs erase it. Include taker or maker fees as appropriate, spread, slippage, latency, funding, gas, and the chance that an exit cannot be filled. Use a conservative fallback when market-depth evidence is missing rather than silently assuming zero cost.
Apply costs at the event where they occur. A single flat deduction can hide the difference between entries, exits, venues, and stressed conditions. Then rerun the test under worse costs and delayed decisions. An idea that survives only the friendliest profile is not robust evidence.
Respect overlapping horizons and repeated trials
Hourly observations with a 24-hour outcome overlap heavily. They are not 24 independent bets. Confidence intervals and test statistics should account for serial dependence. Block bootstrap methods preserve chunks of the time series; heteroskedasticity-and-autocorrelation-consistent inference adjusts for correlated errors.
Report the interval, not only the estimate. If an information coefficient is positive but its interval spans zero, the honest result is inconclusive. Correct for repeated testing when many related variants compete for attention.
What an honest negative result looks like
Crypto Signal Lab published a time-series momentum benchmark using 22,536 contiguous hourly bars for each of BTC, ETH, and SOL from January 2024 through July 2026. The analysis used 2,000 dependence-aware bootstrap resamples and a cost proxy that included fees, spread, and market impact.
| Published fact | Result | Interpretation |
|---|---|---|
| Asset-horizon cells | 9 | BTC, ETH, and SOL across three horizons |
| Mean net episode returns below zero | 8 of 9 | The tested formulation did not show broad positive economics |
| 95% intervals crossing zero | 9 of 9 | No cell established a reliable directional relationship |
| Production claim | None | The result remains negative evidence, not a promoted edge |
The useful conclusion was not to hide the test or tune until it passed. It was to publish that this formulation had not earned authority. The full equations, evidence bundle, and caveats are available in the public research methodology.
Backtest checklist
- Was the signal definition frozen before the result was inspected?
- Could every input have been known at the historical decision time?
- Are the evaluation folds ordered through time?
- Are overlapping labels isolated across split boundaries?
- Is the final holdout genuinely untouched?
- Do costs reflect venue, horizon, latency, and fill risk?
- Does inference account for serial dependence?
- Are repeated trials and parameter searches recorded?
- Are negative and unfillable cases retained?
- Can another researcher reproduce the result from the frozen inputs?