Overfitting

Bailey, Borwein, López de Prado and Zhu describe backtest overfitting as the effect of trying many variations on the same history until one looks successful, then treating that variation as if it had been chosen in advance. Their Notices of the American Mathematical Society paper is about that selection effect. Changing a length or a threshold after seeing the only sample is a small version of the same problem. Fixing the settings before the run, and not searching them, means that search was not done. It does not show that the fixed settings will behave the same way on later prices.