A backtest without a control is a story, not evidence
Someone shows you a chart. A rule was switched on two years ago, and the line goes up and to the right. The rule, you are told, works.
Maybe. But there is a question that chart cannot answer, and it is the only question that matters: compared to what?
The rising-tide problem
Over the two years in that chart, the asset itself may have doubled. If it did, then almost anything that held the asset for most of that time also went up. A rule that bought on Mondays. A rule that bought when the moon was full. A rule that bought whenever the founder’s cat walked across the keyboard.
None of those rules “work”. They were simply present while the tide came in.
This is the most common way a backtest misleads, and it misleads honest people, including the person who built it. The line going up is real. The conclusion that the rule made it go up is the part nobody checked.
What a control is
A control is the most boring possible version of the same experiment. Same asset, same period, same total money, but with the clever part removed.
- Testing a rule that adds more when the price dips? The control is plain scheduled spending of the same total, on the same dates, with no dip logic.
- Testing a rule that enters on a signal? The control is entering at random times, the same number of times, holding for the same length of time.
- Testing a rule that sits out “bad” periods? The control is simply holding throughout.
The control is not a competitor you are trying to beat for bragging rights. It is the ruler. Without it, you have one number and nothing to hold it against.
Compare the right number
Once there is a control, the next mistake is comparing the wrong thing.
Say a dip-adding rule finished with a larger final value than plain scheduled spending. It probably also spent more money, and spent it earlier. In a period where the price rose, money that went in earlier finishes ahead, whatever logic put it there. The final value is rewarding the timing of the spending, not the rule.
The fair comparison holds the total spent equal and asks a narrower question: what did each approach pay per unit? If the rule’s average cost per unit is meaningfully lower than the control’s, over the same spend, the rule did something. If the two are within a whisker of each other, the rule added complexity and nothing else, and that is worth knowing too.
Three checks before believing any backtest
- Is there a baseline beside it? If the only line on the chart is the strategy, stop there. Ask for the boring version.
- Was the same money deployed? A result that needed more capital, or needed it sooner, is not comparable until that is equalised.
- Does it survive a different window? Run six months, twelve and twenty-four. A rule that only looks good in one of them has taught you about that window, not about the rule.
A fourth, for the careful: count the trades. Thirty trades is an anecdote. A conclusion drawn from thirty coin flips would not convince you either.
What a backtest still cannot tell you
Even a backtest with a proper control is a study of prices that already happened. It assumes every order filled at the price on the chart. It usually ignores fees, slippage and the minimum order sizes an exchange enforces. And it knows nothing about the next twelve months, because nothing does.
A controlled backtest is for eliminating ideas: it is very good at showing that a rule did nothing. It is much weaker at proving a rule will keep doing something. Treat a pass as “not yet ruled out”, never as a promise.
Try it yourself
The replay lesson runs a schedule over 24 months of real daily prices and always shows plain scheduled spending of the same money beside it, for exactly the reason above. No account is needed. If a term here was unfamiliar, the glossary defines it in plain language.
Educational content only. Stralines Labs Private Limited is a software company. Nothing here is investment advice. Crypto markets are highly volatile and trading can be risky; past prices do not predict future results.